Data Quality & Filtering

Define the release contract before choosing filters

A release contract states what the dataset is for and what “eligible” means in that context. It should name:

• Intended training and evaluation uses, prohibited uses, and downstream owners
• Allowed sources, licenses or usage bases, privacy rules, deletion obligations, and retention limits
• Required languages, domains, formats, time ranges, and minimum coverage by important slice
• Hard exclusions that block release and soft signals that require calibration or review
• Error budgets for false rejection and policy leakage, including who accepts each risk
• Minimum audit sample sizes, reviewer guidance, disagreement handling, and escalation paths

Order a cascade with measurements

An early-exit cascade avoids expensive work after a record has already failed. For a fixed deterministic policy:

1. Each record enters the first gate.
2. A failed record is quarantined with the first-failure reason and skips later gates.
3. A passing record moves to the next gate.
4. Only records that pass every gate enter the candidate release.

The lab computes two quantities directly:

• Stage CPU = records evaluated at that stage × measured milliseconds per record
• Total fixture CPU = sum of stage CPU across the cascade

Gate order can change compute and first-failure reason counts when records fail more than one gate. It must not change eligibility if every gate is deterministic, uses the same inputs, and “reject if any gate fails” remains the policy. Stateful models, shared rate limits, or gates that mutate content break that assumption and require a different experiment.

Calibrate thresholds against both error directions

A model-backed gate needs reviewed examples from the intended population. For a quality threshold that accepts records at or above a score:

• False rejection means a human-eligible record was rejected. It measures useful coverage lost.
• Policy leakage means a human-ineligible record was accepted. It measures failures inside the published set.
• Uncertain means the available evidence is insufficient for automatic disposition. It should be routed, not hidden.

Evaluate rates by important slice as well as overall. Language, source type, domain, time period, content length, and demographic context can change score distributions and error costs. A threshold that clears a global average can still fail a smaller slice badly.

Treat reviewed slices as release evidence

The threshold lab uses a fixed synthetic audit fixture. For each selected slice:

1. Records with scores at or above the threshold are predicted eligible.
2. Automated decisions are compared with the fixture's human-reviewed eligibility label.
3. False-rejection rate divides rejected eligible records by all eligible records.
4. Leakage rate divides accepted ineligible records by all accepted records.
5. The example release passes only when both declared guardrails hold.

These rates answer different questions and use different denominators. Report the counts with the rates so reviewers can see when a small sample makes a percentage unstable. Production evidence also needs sampling design, reviewer agreement, uncertainty estimates, and documented adjudication.

Use toxicity, privacy, and bias signals carefully

Safety and responsibility are policy domains, not interchangeable classifier labels.

Toxicity and harmful content

• Define the prohibited behavior and intended use before selecting a model or threshold.
• Measure false positives and false negatives by relevant language and context.
• Distinguish quoted, counterspeech, educational, clinical, and targeted-abuse contexts where policy requires it.
• Preserve reviewer escalation for high-impact or ambiguous cases.

Privacy and sensitive data

• Minimize collection before attempting detection or redaction.
• Separate access control, legal basis, consent, retention, deletion, and de-identification from content-quality scoring.
• Test detectors on the actual formats and languages in the source snapshot.
• Treat unresolved sensitive-data uncertainty as a release blocker where policy requires it.

Representation and harmful bias

• Measure which sources, languages, communities, and viewpoints are removed at every stage.
• Audit false rejection and leakage by relevant slices and intersections.
• Investigate whether deduplication or boilerplate rules disproportionately erase repeated minority-language or community forms.
• Document residual limitations; “bias removed” is not a defensible release claim.

Publish policy, evidence, and lineage together

1. 1

   Contract

   Version the policy

   Freeze eligibility rules, thresholds, model and parser hashes, review guidance, protected exceptions, and release owners.

2. 2

   Execution

   Run and reconcile

   Process an immutable source snapshot, verify counts across every disposition, and keep quarantined records access-controlled.

3. 3

   Evidence

   Audit decisions

   Sample accepted, rejected, and uncertain records; report error directions, reviewer agreement, and slice coverage.

4. 4

   Artifact

   Sign the release

   Bind shard hashes, manifests, policy identity, reason counts, audit results, known limits, and deletion procedures.

Minimum release artifacts

• Dataset and shard identifiers with cryptographic hashes
• Source snapshot identity, collection window, provenance, and usage restrictions
• Policy version plus parser, deduplication, classifier, and threshold versions
• Counts and rates by disposition, reason code, source, language, and important slice
• Human-review sampling plan, guidance, disagreement, adjudication, and findings
• Known blind spots, intended uses, prohibited uses, owners, retention, and deletion process

Monitor decisions, not one quality score

When a metric moves, identify which input changed:

• Source mix, collection window, or upstream parser
• Policy, threshold, model, feature, or normalization version
• Reviewer pool, guidance, sampling frame, or adjudication process
• Infrastructure errors, timeouts, missing features, or silent fallbacks

Pass rate alone cannot distinguish these causes. Keep version dimensions in telemetry and sample the changed decision buckets before modifying policy.

Production review checklist

Before execution

• Freeze the source snapshot and release policy.
• Verify source eligibility, privacy handling, deletion obligations, and access controls.
• Benchmark gate cost and quality on representative payloads rather than copying fixture values.
• Create reviewed audit slices for accepted, rejected, and uncertain decisions.

Before publication

• Reconcile input, publish, quarantine, error, and dropped counts exactly.
• Confirm every decision has a reason code and component versions.
• Require slice-level error budgets and coverage floors, not only corpus-wide averages.
• Test replay with the same policy and controlled reprocessing with a new policy.

During operation

• Alert on missing telemetry, parser failures, unexpected source mix, decision-rate shifts, and audit regressions.
• Keep quarantined data isolated, access-controlled, and subject to retention and deletion policy.
• Require review before threshold changes and preserve before-and-after evidence.
• Rehearse dataset withdrawal and downstream notification.

Primary references

• TensorFlow Data Validation: schema, anomaly, skew, and drift checks
• NIST AI RMF Generative AI Profile
• Data Cards: Purposeful and Transparent Dataset Documentation
• Deduplicating Training Data Makes Language Models Better