Web-Scale Data Sources
Design pretraining data mixtures with provenance, licensing, deduplication, contamination controls, quality filters, and release evidence.
What is a pretraining data source?
A pretraining data source is an upstream collection of documents, code, media, or records that may contribute examples to a foundation model's first large training phase. Web crawls, encyclopedias, books, repositories, research papers, licensed archives, and first-party data are source families.
A source is not yet a training dataset. Before a byte can become a training token, the team must establish permission and provenance, extract documents, measure quality, remove sensitive or unsafe material, deduplicate repeated text, protect evaluation sets, choose mixture weights, and publish a reproducible corpus version.
Publicly reachable does not mean licensed, consented, private-data-free, representative, or suitable for training. Governance starts when data is acquired, not after the model is trained.
Define the corpus contract before collecting bytes
A corpus contract turns "get more data" into testable requirements:
- Purpose: name the model capabilities and languages the corpus is meant to support.
- Permission boundary: record license, contract, consent, opt-out, retention, and geography constraints per snapshot.
- Quality boundary: define measurable acceptance rules and review samples for both accepted and rejected documents.
- Evaluation boundary: reserve benchmark families and private test sets before training data is assembled.
- Reproducibility boundary: version source snapshots, extraction code, classifier checkpoints, thresholds, and mixture weights.
1 document ID
Lineage key
Connect every token shard to a source snapshot and curation decision.
2 review samples
Filter evidence
Inspect accepted and rejected examples for every important slice.
0 benchmark matches
Release target
Block known exact and reviewed near-matches before training.
Different source families contribute different evidence
Breadth
Broad web
Large topical and language coverage, with substantial boilerplate, duplication, provenance uncertainty, and uneven representation.
Structure
Curated reference
Attributable, edited material with strong structure, but less variety in voice, recency, and domain coverage.
Formal patterns
Code and technical
Syntax and technical reasoning with license, secret, vulnerability, generated-code, and repository-fork controls.
Governance
Licensed and first-party
Clearer rights and metadata, though privacy, consent, quality, and representation still require review.
The design question is not "which source is best?" It is "which measured mixture satisfies this model's capability and governance contract?"
Turn snapshots into a versioned corpus
1 Acquire
Register and quarantine
Record origin, retrieval time, permission basis, checksums, retention policy, and a stable source ID before processing.
2 Parse
Extract and classify
Decode formats, preserve metadata, segment documents, identify language, and quarantine malformed or encrypted payloads.
3 Filter
Curate and protect
Apply quality, safety, privacy, deduplication, opt-out, and evaluation-overlap gates with explicit decision reasons.
4 Release
Mix and publish
Choose source weights, tokenize deterministically, emit slice statistics, sign the manifest, and freeze the complete recipe.
Choose the Common Crawl representation deliberately
Common Crawl publishes regularly collected web archives at petabyte scale. Its formats expose different points in the extraction pipeline:
Raw archive
WARC
Stores raw crawl requests, responses, headers, payloads, and crawl metadata. Choose it when extraction fidelity, provenance, or non-text content matters.
Extracted text
WET
Stores extracted plaintext plus basic WARC metadata. It is faster to consume, but the upstream extraction decision is already baked in.
Metadata and links
WAT
Stores computed JSON metadata, HTTP headers, and links. Use it for crawl analysis, graph signals, and provenance joins without downloading every body.
WET is convenient, not automatically clean. Boilerplate visible in Common Crawl's own WET example shows why extraction quality still needs measurement.
Make every curation gate auditable
Quality filtering and deduplication are model decisions, not neutral cleanup:
- A global quality threshold may reject dialects, code-switching, short reference answers, or smaller languages at a higher rate.
- Exact hashes catch byte-identical copies; normalized hashes catch formatting variants; MinHash or embedding-based methods can find near-duplicates at higher cost and review burden.
- Aggressive near-deduplication reduces repeated exposure but can collapse legitimate templates, translations, and rare variants.
- Evaluation decontamination must compare against held-out content before the training split is finalized.
The invariant is stronger than "the filter ran": accepted plus rejected counts must equal input counts, and every decision must be reproducible from a versioned rule or model.
Evidence from open web-corpus work
- The FineWeb paper documents a 15-trillion-token corpus built from 96 Common Crawl snapshots and ablates extraction, filtering, and deduplication choices instead of treating one recipe as universal.
- The maintained FineWeb dataset card now describes more than 18.5 trillion cleaned and deduplicated English tokens, which is a reminder to pin dataset revisions rather than cite a moving name alone.
- Lee et al. found that deduplication reduced verbatim memorized output by about tenfold in their experiments and exposed train-validation overlap that could inflate evaluation.
- The NIST AI Risk Management Framework treats provenance, documentation, third-party data risk, and ongoing measurement as governance work across the system lifecycle.
These results justify experimentation and documentation. They do not supply thresholds that can be copied unchanged into a different corpus.
Release a corpus only with operational controls
- Keep immutable manifests and aggregate statistics for every training run.
- Preserve document-level lineage through normalized text, dedup clusters, token shards, and model runs.
- Scan for personal data, credentials, malware, and unsafe executable content before durable storage.
- Make opt-out and deletion workflows locate derived copies, not only the original URL.
- Track acceptance, rejection, duplicate, and language distributions by source and over time.
- Re-run contamination checks whenever the evaluation inventory or corpus recipe changes.
- Require a signed review for licenses, high-risk sources, unexplained distribution shifts, and missing lineage.