ToxiGen & HateCheck: Bias Detection

What are ToxiGen and HateCheck evaluations?

ToxiGen and HateCheck are complementary test resources for group-targeted harmful-language detection. ToxiGen supplies broad, adversarial examples of implicit harm and benign identity mentions. HateCheck supplies controlled pairs that isolate one linguistic behavior at a time, such as negation, counter-speech, or spelling variation.

They matter because one aggregate accuracy score can hide two costly errors: missing subtle harm and blocking legitimate speech about protected groups. Invariant: a safety claim must report both harmful-content recall and legitimate-content false positives by decision-relevant slice. Neither benchmark alone certifies a production moderation system.

Use the benchmarks for different questions

Broad adversarial coverage

ToxiGen

The original dataset contains 274,186 machine-generated toxic and benign statements about 13 minority groups. Its construction emphasizes implicit language and reduces the shortcut that treats every identity mention as toxic.

Controlled diagnosis

HateCheck

The original English suite contains 3,728 cases across 29 functionalities in 11 classes. Minimal contrasts reveal whether one linguistic change flips behavior for the right reason.

Representative evidence

Product challenge set

Versioned, consented or appropriately governed examples reproduce the product's languages, policies, conversation context, appeals, and user consequences.

Ambiguous policy

Human adjudication

Qualified reviewers resolve unclear cases, calibrate automated evaluators, document disagreement, and protect reviewer wellbeing around sensitive material.

Keep the claim bounded

ToxiGen asks: can the system separate implicit harmful statements from benign statements that mention the same groups?
HateCheck asks: which defined linguistic functions does the classifier handle or fail?
A product suite asks: do those behaviors transfer to the deployed policy, language, context, and traffic mix?
Operations ask: can reviewers, appeals, monitoring, and rollback contain the remaining errors?

Understand how ToxiGen creates broad stress coverage

1
Coverage
Define identity groups
The authors selected 13 groups in a United States sociocultural context and sought both toxic and benign statements for each group.
2
Adversarial
Generate difficult examples
Demonstration prompts and classifier-in-the-loop decoding guided GPT-3 toward subtle examples that were difficult for an existing detector.
3
Shortcut control
Create a balanced corpus
Roughly equal toxic and benign generation reduces the spurious association between a group identifier and the toxic label.
4
Human review
Validate a challenging subset
Human evaluation checked a selected difficult subset. Do not describe every generated row as independently human-adjudicated evidence.

The paper reports that 98.2% of its statements contain no explicit profanity and that human annotators labeled 94.5% of the evaluated toxic examples as hate speech. Those figures describe the authors' dataset and protocol; they are not expected production rates.

Record provenance before scoring

Dataset release, split, row ID, content hash, generation metadata, and license terms.
Human-label source and agreement status; keep model-generated labels distinct from adjudicated labels.
Target-group and benign-versus-harmful strata, with intersectional or multilingual gaps stated explicitly.
Model, tokenizer, prompt, classifier, threshold, preprocessing, and policy versions.
Raw score, binary action, reviewer outcome, appeal result, and final adjudication.

Choose a threshold without hiding the harmed slice

A classifier score becomes a moderation action only after a threshold is chosen. Lowering the threshold usually catches more harmful cases but can overflag benign identity mentions, counter-speech, and reporting. Raising it can reduce reviewer volume while missing implicit or obfuscated harm.

Change the candidate, threshold, and daily volume below. The lab recomputes recall, false-positive rate, the worst harmful slice, reviewer load, and a bounded rollout decision from one synthetic evidence table.

Threshold and slice lab

Move the threshold and watch both errors move

Choose a synthetic candidate, set its operating threshold, and scale daily traffic. The release decision checks harm, legitimate use, worst-slice behavior, and reviewer capacity separately.

Loading threshold evidence

Read the operating point as a set of consequences

Harm recall counts detected harmful cases over all harmful cases.
Legitimate false-positive rate counts benign cases sent to the harmful path over all benign cases.
Worst-slice miss rate prevents an easy, high-volume slice from masking failure on implicit or perturbed harm.
Review demand turns the benchmark operating point into an operational capacity requirement.
The release gate remains provisional because these are synthetic teaching observations, not a product's validation result.

Use HateCheck to isolate a behavior

HateCheck follows functional software testing: define a behavior, construct cases that should pass, and inspect failures by function rather than averaging them into one number. Its 29 functionalities include 18 hateful and 11 contrastive non-hateful tests.

Should detect

Expression

Derogation, threats, slurs, profanity used as hate, and hate expressed through reference or implication.

Should not detect

Contrastive non-hate

Positive identity statements, non-protected targets, counter-speech, quoted discussion, and reclaimed terms under the suite's labels.

Should stay stable

Perturbation

Negation, spelling variation, phrase splitting, and pronoun reference test whether surface changes break the intended decision.

The original suite is English, text-only, document-level, and limited to cases with a clear gold label under its protocol. It does not establish performance for images, conversations, every culture, every policy definition, humor, or coded language that requires external context.

Predict the shortcut before revealing the contrast

Choose a candidate and a functional pair, then predict the failure pattern. The result reveals both decisions together because a detector that catches the harmful case but blocks its minimal non-hateful contrast has not passed the function.

Functional contrast audit

Predict which minimal change breaks the detector

Choose one candidate and one controlled function. Commit a prediction before revealing both case-level decisions and the engineering diagnosis.

Loading functional contrasts

Turn a functional failure into an engineering action

Negation failure: inspect context encoding and add independently held-out negated contrasts; do not add a brittle word exception.
Counter-speech overblocking: preserve quotation and speech-act context through preprocessing, policy labeling, and reviewer tools.
Non-protected target overblocking: separate general negative sentiment from group-targeted harm.
Spelling-variation miss: test normalization and character-aware features against clean-text regressions and adversarial adaptations.

Build one reproducible evaluation path

Group-targeted harm evaluation path

The release record keeps broad coverage, functional diagnosis, and product evidence separate.

Evidence

Versioned cases

ToxiGen, HateCheck, product cases, provenance, labels, slices, and protected holdouts.

System under test

Candidate scorer

The exact preprocessing, model, threshold, policy mapping, and escalation behavior proposed for release.

Measure

Slice diagnosis

Confusion counts, uncertainty, target-group slices, functional pass rates, and independently reviewed disagreements.

Decide

Bounded rollout

Predeclared gates, review capacity, canary exposure, appeals, monitoring, and rollback ownership.

Do not collapse the report to one F1 score

Report confusion counts, precision, recall, false-positive rate, and denominators at the declared threshold.
Add confidence intervals for overall and decision-critical slice rates.
For HateCheck, report accuracy per functionality and the number of functions with complete pass, partial pass, or no reliable evidence.
For ToxiGen, separate toxic and benign performance by group and by implicit-versus-explicit characteristics available in the release.
Review disagreements and high-severity misses; a weighted average must not offset a critical failure with an unrelated easy slice.

Implement slice and function gates with auditable inputs

The example uses only the Python standard library. It validates records, computes confusion metrics at a frozen threshold, reports harmful-slice miss rates and functional accuracy, and keeps hard release gates separate.

Slice-aware ToxiGen and HateCheck evaluation

Production systems should read immutable prediction artifacts rather than call a live model during scoring. Persist the exact case set, prediction file, policy, threshold, evaluator code version, and release decision so another reviewer can reproduce the result.

Operate the benchmark as a governed safety process

Before a release

Freeze the threshold and slice gates on a calibration partition before opening the protected release set.
Deduplicate exact and semantic neighbors across training, prompt development, calibration, and release partitions.
Add product-specific languages, context lengths, policy boundaries, and traffic weights that the public suites do not cover.
Adjudicate ambiguous and high-severity cases with trained reviewers and document disagreement.
Capacity-test human review, appeals, and rollback at the expected flagged volume.

After a release

Monitor harmful misses, legitimate overblocks, appeals, reviewer reversals, and time-to-resolution by slice.
Sample from real product paths under appropriate privacy and data-governance controls.
Convert confirmed incidents into a future protected regression set, not an immediately reusable tuning set.
Revalidate after model, prompt, normalization, policy, language, or threshold changes.
Publish limitations and avoid claiming universal hate-speech detection from two English-centric public resources.

ToxiGen provides breadth and HateCheck provides diagnosis. Product evidence provides relevance. A defensible release needs all three, plus human and operational controls for the errors that remain.

Read the original protocols and limitations

Use the papers and accompanying dataset documentation as the source of truth for construction, labels, splits, and permitted claims. Recheck licenses, content warnings, and data-handling requirements before downloading or redistributing sensitive examples.

No quiz questions available

Could not load questions file