Advanced Safety Evaluation

Start with a harm model, not a benchmark name

A harm model describes the outcome the system must prevent or bound. It connects an affected party, an unsafe behavior, an enabling system path, and a consequence. Benchmarks are evidence sources; they do not define the product's risk by themselves.

For each harm, record:

• Affected party: the user, a subject mentioned in content, an operator, a bystander, or the public.
• Unsafe behavior: harmful assistance, discriminatory treatment, privacy disclosure, manipulation, misinformation, or unsafe tool action.
• System path: direct chat, retrieval, memory, file upload, tool output, multi-turn state, or post-processing.
• Severity and exposure: how bad one event is and how often the path can occur.
• Required response: refuse, safely redirect, request confirmation, remove a capability, escalate to review, or block release.

Estimate the evidence matrix before collecting cases

Safety test volume grows across independent dimensions. Suppose a release must cover 6 harm families, 4 languages, 3 system modes, and 2 severity bands:

6 × 4 × 3 × 2 = 144 distinct cells.

At 50 reviewed cases per cell, the initial matrix contains 144 × 50 = 7,200 cases. This is a planning estimate, not a universal minimum. It exposes where the budget goes and prevents a large easy slice from hiding an empty critical cell.

Allocate more cases to high-severity, low-frequency paths and to slices with wide uncertainty. Equal counts are easy to explain, but risk-weighted sampling is usually more defensible.

Build a reproducible evaluation system

The runner must exercise the deployed application, not only a base model in isolation. A safe answer from the model does not validate retrieval authorization, tool permissions, or an output filter configured with a different policy version.

Construct datasets that support a release claim

1. 1

   Provenance

   Source cases

   Combine public benchmarks, domain-expert cases, consented and minimized production traces, red-team attacks, and prior incidents.

2. 2

   Policy

   Label behavior

   Record expected policy, severity, allowed assistance, refusal quality, affected slice, rationale, and reviewer confidence.

3. 3

   Integrity

   Partition and deduplicate

   Separate build, calibration, release, and hidden sets; detect exact and semantic near-duplicates across every split.

4. 4

   Control

   Version and review

   Freeze the release set before scoring candidates and require approval for additions, relabeling, threshold changes, or exceptions.

Give every case enough context

A useful case record includes:

• Input prompt plus conversation history, retrieved content, tool results, locale, and relevant user state.
• Expected policy behavior and a severity label, not only a binary safe/unsafe answer.
• Slice tags such as language, region, age-related context, domain, attack family, capability, and system mode.
• Source, consent or lawful basis where relevant, retention class, and access restrictions.
• Dataset version, split, duplicate cluster, labelers, adjudication status, and change history.

Protect release-set independence

Leakage occurs when release cases, close paraphrases, expected answers, or judge rationales influence tuning or prompt development. Control it with exact hashing, semantic-neighbor review, source-level quarantine, hidden holdouts, and access logs.

Do not solve leakage by deleting provenance. A trustworthy team must be able to explain where each case came from and why it belongs in the release set.

Use public benchmarks as seeds, not certificates

Public datasets make methods reproducible and provide shared baselines, but their distributions rarely match one product exactly.

Human preference data such as arena votes can reveal over-refusal and practical usefulness, but preference is not proof of harmlessness. Analyze safety slices separately and account for who supplied the prompts and votes.

Report confidence and slices, not only point estimates

An observed failure rate is incomplete without its denominator and uncertainty. Zero failures in a small sample does not mean the true rate is zero.

For zero observed events, the rule of three gives a quick approximate 95% upper bound: 3 / n.

• 0 / 30 cases supports only an upper bound near 10%.
• 0 / 300 cases supports an upper bound near 1%.
• 0 / 3,000 cases supports an upper bound near 0.1%.

Use a binomial interval such as Wilson for routine proportions, and predeclare how the release gate uses the upper bound. For clustered users, repeated attacks, or adaptive tests, independence assumptions can fail; use grouped analysis or bootstrap at the user, session, or campaign level.

Slice the dimensions that can change the decision

At minimum, inspect harm family, severity, language, region, user group, domain, system surface, conversation length, attack family, and evaluator confidence. Report both the overall result and the worst decision-relevant slices.

A slice gate needs enough evidence to be interpretable. If the upper confidence bound is too wide, the correct result is insufficient evidence, not pass.

Exercise a multi-axis release gate

Change the evidence bundle, sample size, severe-harm ceiling, and requested exposure. Watch how confidence, slice quality, refusal cost, privacy, leakage, label reliability, and rollout stage remain separate decisions.

Calibrate evaluators against human judgment

No evaluator is a neutral oracle. Each method has a different failure pattern.

Calibrate on a separately labeled set:

1. Have qualified reviewers label cases independently and adjudicate disagreements.
2. Compare each automated evaluator with the adjudicated labels by slice and severity.
3. Measure false-negative rate, false-positive rate, precision, recall, and calibration at the proposed threshold.
4. Route low-confidence, novel, and high-severity cases to human review.
5. Version the evaluator model, prompt, rubric, examples, threshold, and dependency configuration.

False negatives expose harmful behavior. False positives can over-refuse legitimate requests, reduce access, and create unequal product quality. Both belong in the release evidence, but severe misses usually receive stricter gates.

Design red teams around attack coverage

An attack campaign needs a declared coverage model: which assets, trust boundaries, attack families, languages, session lengths, and attacker budgets it exercises.

A large prompt count can still provide weak coverage when all prompts are close variants of one public jailbreak. Stronger campaigns combine:

• Known regressions for reproducibility.
• Taxonomy-driven mutations across harm families and system surfaces.
• Human creativity from people with relevant language, cultural, domain, and security expertise.
• Automated search that adapts to accept/reject signals within an explicit query budget.
• Multi-turn, retrieval, memory, file, and tool scenarios that exercise the full application.
• Independent adjudication of attack validity, policy outcome, severity, reproducibility, and mitigation status.

Track two denominators separately: the fraction of declared attack paths exercised and the attack success rate within those paths. A low success rate over narrow coverage is not strong evidence.

Plan a red-team campaign and expose its blind spots

Change the deployed surface and campaign design. Then add languages, multi-turn sessions, and mixed human review. The visible matrix shows why direct jailbreak replay does not cover indirect prompt injection, tool misuse, or cumulative privacy attacks.

Implement a slice-aware release decision

The example computes Wilson upper bounds for each rate, keeps critical and non-critical gates separate, and blocks evidence with privacy leaks, holdout contamination, or unreliable labels.

Production code should also persist the dataset, system, evaluator, policy, and threshold versions that produced each result. A Boolean without provenance is not an auditable release decision.

Diagnose common evaluation failures

Make trade-offs explicit

Safety thresholds move consequences; they do not eliminate them.

• Stricter harmful-content thresholds can reduce unsafe releases while increasing false refusals and unequal access for dialects or sensitive domains.
• Broader datasets improve coverage but raise labeling cost, privacy exposure, reviewer burden, and maintenance work.
• Hidden holdouts protect independence but reduce day-to-day debugging access and require stronger governance.
• Automated red teaming increases search breadth but can overfit to the attacker model and produce invalid or unrealistic cases.
• Human red teaming adds context and creativity but requires specialist recruitment, secure handling, reviewer wellbeing controls, and careful adjudication.
• Larger canaries produce production evidence faster but increase blast radius when a gate or monitor is wrong.

Resolve these trade-offs through risk-tiered gates and bounded exposure. Do not hide them inside one weighted score whose weights can silently trade a critical failure for an unrelated improvement.