ML Datasets: Deep Dive

Start with a dataset contract

A dataset contract is a short, testable description of what data may enter a model and what evidence must travel with it. Write it before collecting data so a large corpus cannot hide an unclear purpose.

Before choosing a source, answer these questions:

• What decision will the model support, and what is the harm when it is wrong?
• Which people, regions, languages, devices, or rare cases need explicit coverage?
• Which rights, privacy, retention, and redistribution constraints apply?
• What unit must remain together when data is split: a user, document family, patient, merchant, or time period?

Turn raw sources into a reproducible corpus

Collection is not just downloading files. A production dataset pipeline preserves evidence about where examples came from and makes each transformation repeatable.

1. 1

   Permission

   Acquire

   Capture source, license, consent or policy basis, retrieval time, and immutable raw-file reference.

2. 2

   Structure

   Normalize

   Extract useful fields, standardize encoding and language, and retain a stable example ID with provenance.

3. 3

   Quality

   Filter

   Remove unwanted formats, personal data, unsafe material, spam, boilerplate, and duplicates with recorded rules.

4. 4

   Evidence

   Freeze manifests

   Create split manifests, compute statistics, version code and thresholds, then make the snapshot reproducible.

Choose sources for the signal they add

• Web crawls provide broad language and long-tail coverage, but need aggressive quality, rights, and duplication controls. Common Crawl archives raw web material in WARC files, extracted text in WET files, and metadata in WAT files.
• Curated reference material such as encyclopedias, documentation, books, and academic papers can provide denser signal and clearer structure. It still needs license and freshness review.
• Community and code data can teach dialogue, troubleshooting, and programming patterns. Quality signals such as accepted answers or repository activity are useful filters, not proof that use is permitted.
• Synthetic or simulated data can fill rare scenarios or expensive labels. Keep it distinguishable from real data and evaluate whether it improves performance on independently collected real examples.

Inspect well-known corpus patterns

Named datasets are useful examples of design choices, not ready-made defaults. Their snapshots, licensing terms, filters, and known limitations change over time; read the dataset card and release manifest for the exact version you intend to use.

Build a mixture that serves the task

Dataset mixing is the weighted combination of source streams during training. A source's raw size does not have to equal its training weight: smaller, trusted material is often sampled more often when it supplies a capability that a large noisy source lacks.

Use the lab to select a target workload, allocate a fixed 100% training budget, and see the capability gaps or overexposure the mixture creates. There is no universally correct ratio; the lesson is to make the intended capability and its trade-off explicit.

Use source-specific controls

• Apply deduplication across sources, not only within each source. A copied document can appear in a crawl, a forum post, and a repository.
• Keep provenance fields through every join and transformation so a source can be audited or removed later.
• Measure coverage by task-relevant slice. A mixture that looks diverse at the domain level can still omit a language, time period, or rare intent.
• Make the sampling manifest immutable for a training run. Record source versions, weights, filtering rules, tokenizer version, and random seed.

Protect evaluation from contamination

Data contamination happens when an evaluation example, or a close enough relative, enters training data. The model may then appear capable because it memorized the answer or context rather than generalizing to unseen inputs.

Try three split policies in the lab. Add duplicate or future-looking records to reveal why random row splitting is sometimes insufficient even when the train, validation, and test percentages look sensible.

Check more than exact matches

Exact hashes catch byte-for-byte copies. Near-duplicate methods such as n-gram overlap, MinHash, SimHash, or embedding similarity catch paraphrases and template variants. For benchmarks, also check:

• Entity leakage: keep one user, patient, merchant, conversation, or document family in one split when their repeated appearances would reveal the answer.
• Temporal leakage: train only on information available before the prediction or benchmark cutoff.
• Benchmark leakage: search training candidates for benchmark prompts, answers, and close variants; remove matches and document the procedure.
• Process leakage: restrict access to final-test labels and results so repeated tuning cannot turn the test set into validation data.

Evaluate a candidate dataset before training

Dataset evaluation turns a vague claim such as "this looks clean" into evidence that a specific snapshot is suitable for a defined use. Start with a small, representative sample and make the decision reviewable before spending on a full training run.

Review the evidence in order

1. Documentation: read the collection method, intended uses, known limitations, license, and version history. Dataset cards are a starting point, not a substitute for policy review.
2. Schema and samples: inspect missingness, encoding, language, labels, personally identifying fields, and failure cases. Sample by source and important slice, not only at random.
3. Statistics and coverage: calculate class balance, length distributions, temporal coverage, duplicate rate, and representation across the chosen population.
4. Pilot evaluation: train a bounded baseline, evaluate on protected holdouts, and compare errors by slice. A score increase can still be unacceptable if it worsens a critical group.
5. Release decision: version the approved manifest with its documentation, filters, quality report, and approval record. Preserve the raw snapshot according to retention policy.