7.1 — The training data: pretraining corpora, deduplication, filtering

1. Why this session exists

We spent Chapter 6 building the architecture — the residual stack, attention, FFN, RoPE, LayerNorm. None of it works without the data. A randomly-initialised 7B decoder-only Transformer produces incoherent noise; the same network trained on 2 trillion tokens of well-curated text becomes LLaMA-2-7B, capable of summarising documents, writing code, and answering questions. The architecture is the kernel; the data is the mass.

This session is the first of the "training" half of the course. It maps the modern pretraining-data landscape:

1. Where the tokens come from — web scrapes, books, code, scientific text — and roughly in what proportions.
2. The pipeline from "raw 100 TB Common Crawl dump" to "1–10 T training tokens": extraction, language ID, quality filtering, deduplication, decontamination.
3. The canonical open corpora — C4 (2019), The Pile (2020), RedPajama (2023), DCLM (2024), FineWeb and FineWeb-Edu (2024) — and what each contributed methodologically.
4. The "data is now the bottleneck" argument that drives the field's recent emphasis on quality over quantity.

After this session you can read a model card's "Training Data" section and recognise the recipe, and you understand why DCLM-trained 7B models in 2024 outperform LLaMA-1 trained on 5× more tokens. We come back to the consumption side — mixing, curriculum, and annealing — in 7.5.

2. Where pretraining tokens come from

A modern LLM is trained on a mixture of sources, weighted by quality and quantity. The classical breakdown, roughly stable across LLaMA, GPT-3, MPT, Mistral, Qwen and DeepSeek:

| Source | What it is | Rough share | Why it's included | |---|---|---:|---| | Web (Common Crawl–derived) | Filtered HTML pages from a years-long crawl of the open web | 60–85% | Sheer breadth of topics, registers, languages | | Books | Project Gutenberg, Books3, Anna's Archive style corpora | 5–15% | Long-form coherence, narrative structure, formal language | | Code | GitHub, BigCode, StackOverflow | 5–20% | Logical/symbolic reasoning, formal syntax, side-effect-free transfer to non-code tasks | | Scientific / academic | arXiv, PubMed, S2ORC | 2–10% | Mathematical notation, technical vocabulary, citation structure | | Wikipedia & reference | Wikipedia (en + multi), Wikidata | 1–5% | High-quality, factual, encyclopedic | | Q&A / instructional | StackExchange, math forums | 1–5% | Question-answer format; useful priming for later SFT |

The Common-Crawl-derived web data is the raw mass — a single year's crawl is 200–400 TB of HTML, mostly noise. The rest are curated sources that are kept in nearly verbatim form. The total token budget after filtering is what we care about; LLaMA-3 (15 T tokens), Llama-3.1 (15 T), DeepSeek-V3 (14.8 T), Qwen2.5 (18 T) are in the 10–20 trillion range.

2.1 Why so much web data?

Two reasons:

(a) Volume. Even after aggressive filtering, the open web is still the only source that delivers $10^{12}$+ unique tokens. Books, code, Wikipedia combined contribute under 1 T tokens at most. Hitting Chinchilla-optimal data (Cap 7.12 (Chinchilla)) for a 70B model requires $\sim 1.4 \cdot 10^{12}$ tokens; for LLaMA-3-405B it's $\sim 8 \cdot 10^{12}$. Only the web can supply that scale.

(b) Diversity. The web is the most linguistically and topically heterogeneous source we have: blog posts, forum discussion, news, technical documentation, recipes, fan fiction, government filings, reviews, classifieds. This breadth is what lets a single LLM answer cooking questions and explain quantum field theory.

The trade-off is quality: most of the web is low-information (SEO spam, navigation chrome, listings, machine-translated content). Hence the entire filtering pipeline of §3.

3. From raw web crawl to training tokens

The pipeline from "raw HTML dump" to "training tokens" is the single biggest engineering effort in pretraining-data work. The standard stages, in order:

3.1 Text extraction

Convert HTML to plain text. Strip navigation, ads, boilerplate (header/footer/sidebar). The Common Crawl Foundation distributes both WARC (raw response) and WET (extracted text) files; modern pipelines re-extract from WARCs with better tools because WET extraction is too lossy. Trafilatura (Barbaresi 2021[12]) is a strong open-source extractor.

3.2 Language identification

Filter to the target language(s). For an English-centric model, drop pages whose detected language is not English. For multilingual models (Aya, BLOOM, mBERT lineage), use language ID to weight languages explicitly. CLD3 and FastText's lid.176 are the standard tools.

3.3 Quality filtering: heuristic

Drop pages that fail simple heuristics:

• Too short (< 50 words) or too long (> 50,000 words).
• Repetitive (top-n-gram fraction > threshold).
• Symbol-heavy (high ratio of non-word characters).
• Lorem-ipsum / placeholder.
• Excessive line-ending characters (extraction artefacts).
• Pages with disallowed phrases (e.g., adult content, spam markers).

Each heuristic catches a specific failure mode. The MassiveText filtering of Gopher (Rae et al., 2021, arXiv:2112.11446)[6] §A.2 codified the now-standard 7-rule list.

3.4 Quality filtering: learned

Train a classifier on a small high-quality sample (Wikipedia, references, etc. as positives; random web as negatives) and use it to rank-filter the web. The CCNet pipeline (Wenzek et al., 2020[7], arXiv:1911.00359) used a perplexity threshold from a small KenLM trained on Wikipedia: keep pages whose Wikipedia-LM perplexity is low (Wikipedia-like) and drop the rest. Modern pipelines (DCLM, FineWeb-Edu) use a fastText classifier trained on hand-labeled examples.

3.5 Deduplication

The single highest-leverage filter. Web scrapes contain massive duplication — the same Wikipedia article quoted on 1000 mirrors, identical CMS-generated boilerplate, copy-paste licence text. Two algorithms:

• Exact (line-level or document-level): hash each unit, drop duplicates. Cheap but misses near-duplicates.
• Near (MinHash + LSH): compute MinHash sketches over n-gram shingles, bucket via Locality-Sensitive Hashing, compute Jaccard similarity within buckets, drop near-duplicates above a threshold (typically 0.8 Jaccard on 5-gram shingles).

Lee et al., 2022, Deduplicating Training Data Makes Language Models Better (arXiv:2107.06499) is the foundational paper. Their result, which now informs every modern pipeline: deduplication consistently improves quality (~2 perplexity points), reduces memorisation (lower verbatim recitation rate), and lets the model converge in fewer steps for the same final loss.

3.6 Decontamination

Remove text that overlaps with evaluation benchmarks: MMLU, HumanEval, GSM8K, etc. If a benchmark question appears verbatim in training data, the model has "seen the answer" and the benchmark loses meaning. Modern pipelines compute n-gram overlap between training and benchmark sets and remove overlapping documents. This is harder than it sounds — paraphrased contamination is hard to detect — but at the very least the exact-match check is done.

3.7 PII removal and toxicity filtering

Optional, recipe-dependent: redact phone numbers, email addresses, social security numbers; classifier-filter toxic content. Some models keep "toxic" content in pretraining and rely on alignment (Cap 9) to suppress harmful outputs; some filter at the pretraining stage. The trade-off is documented in Welbl et al., 2021[11].

3.8 The result

Starting from a $\sim 400$ TB Common Crawl year, the pipeline outputs roughly:

That is one year of crawl. To reach the 10–20 T token regime, modern pipelines stitch together multiple years of CC and then add code, books, and scientific corpora on top.

4. Canonical open corpora

A short tour of the corpora you'll see cited in model cards and ablation studies. Each contributed something methodologically; together they document the field's evolution toward "quality over quantity."

4.1 C4: Colossal Clean Crawled Corpus (2019)

Raffel[1] et al., Exploring the limits of transfer learning with a unified text-to-text transformer (T5) (arXiv:1910.10683) §3.1. ~750 GB, ~150 B tokens, extracted from one Common Crawl snapshot. Heuristic-only filtering: drop pages with placeholder text, profanity, fewer than 5 sentences, etc.

C4 was the first publicly-released, large-scale, quality-filtered web corpus and powered T5. It is the baseline against which every later corpus is measured. Variants: C4 (multilingual), C4-NoClean (no quality filter), C4-RealNewsLike.

4.2 The Pile (2020)

Gao[2] et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling (arXiv:2101.00027). 825 GB, ~300 B tokens, 22 sources mixed by hand: Common Crawl, Wikipedia, books, code, arXiv, PubMed, FreeLaw, government filings, Stack Exchange, Hacker News. This was the first corpus that deliberately mixed many high-quality non-web sources with the web — proving that careful source curation matters.

GPT-J, GPT-NeoX, and Pythia were trained on The Pile. The Pile remains the single best-documented and most-cited open corpus.

4.3 RedPajama (2023)

Together AI[3], RedPajama: An Open Source Recipe to Reproduce LLaMA Training Dataset (togethercomputer.com/redpajama). Open reproduction of the LLaMA-1 data mixture: CommonCrawl-CC-Net (filtered with CCNet), C4, GitHub, books, arXiv, Wikipedia, StackExchange. ~1.2 T tokens. This was the first open-source corpus large enough to retrain a LLaMA-1-class model.

4.4 DCLM: DataComp for Language Models (2024)

Li[4], Fang, Smyrnis, et al., DataComp-LM: In search of the next generation of training sets for language models (arXiv:2406.11794). DCLM was the first paper to make the data-curation step itself the experimental subject. They built a 240 T-token raw pool from Common Crawl and ran controlled experiments on filtering recipes — what classifier, what threshold, what deduplication strategy — measuring downstream loss at fixed compute. The standardised benchmark drove dataset quality forward.

Their best recipe, DCLM-Baseline (~3.8 T tokens), trains 7B models that match LLaMA-2-7B's performance with $\sim 5\times$ fewer tokens. This was a decisive empirical demonstration that better data is more valuable than more data (modulo Chinchilla optimality — Cap 7.12 (Chinchilla)).

4.5 FineWeb and FineWeb-Edu (2024)

Penedo[5] et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (HuggingFace blog, arXiv:2406.17557). 15 T-token web corpus from 96 Common Crawl snapshots, carefully deduplicated and filtered.

FineWeb-Edu is a 1.3 T-token subset filtered for educational quality using a fastText classifier trained on ~500K LLM-annotated examples. Educational web pages (tutorials, how-tos, encyclopedic explanations) are kept; spam, low-content listings, and noise are dropped. Models trained on FineWeb-Edu beat models trained on raw FineWeb at the same token budget by 5–10% on knowledge-intensive benchmarks.

The methodological contribution: classifier-filtering with LLM-annotated training data is now the default high-quality recipe. DCLM and FineWeb-Edu converged on essentially the same approach independently.

4.6 Closed corpora

Frontier labs (OpenAI, Anthropic, Google, Meta) generally do not publish their training data. Some details leak:

• GPT-3 (2020): 300 B tokens — CommonCrawl, WebText2, Books1, Books2, Wikipedia. Mixture ratios are in the paper §2.2.
• LLaMA-3 (2024): 15 T tokens, ~5% non-English, ~15% code; specifics in the LLaMA-3 paper (arXiv:2407.21783) §3.
• DeepSeek-V3 (2024): 14.8 T tokens; details in the DeepSeek-V3 tech report.

The open community typically replicates frontier-data quality at within ~10% of closed-model performance, as DCLM and FineWeb-Edu have shown.[13]

5. The "data wall" and quality over quantity

A central tension in 2024–2026 pretraining:

• The data wall. The supply of unique high-quality text on the open web is roughly fixed. Current estimates put it at $10^{13}$–$10^{14}$ tokens (Villalobos et al., 2022, arXiv:2211.04325[9], "Will we run out of data?"). Once a single training run consumes most of it, further scaling on the data side has diminishing returns.
• Quality > quantity. DCLM and FineWeb-Edu show that better-filtered data beats more data at fixed compute. A 7B model trained on 4 T DCLM-Baseline tokens outperforms one trained on 10 T plain-CC tokens.
• Synthetic data. Future training runs increasingly use LLM-generated synthetic data — pre-existing models generating curated examples for the next generation. This is starting to dominate the post-training regime (instruction tuning, reasoning trajectories) and increasingly bleeds into pretraining (e.g., Phi-3.5's "textbooks are all you need" recipe).

The headline: the current bottleneck is data quality, not data quantity. This is why the entire pipeline of §3 — and especially the learned-classifier filter of §3.4 — has become the centre of pretraining work.

6. Deduplication, in slightly more detail

Because it is the highest-leverage step in §3, deduplication deserves a closer look.

6.1 The problem

Common Crawl contains massive duplication for structural reasons: pages mirror each other, news wires syndicate the same article to thousands of outlets, CMS templates generate near-identical pages with one varying field, browser-history URLs return the same page with different query strings. The raw web is roughly 5–10× duplicated.

Training on duplicated data has three harms:

1. Wasted compute. Each duplicate seen is one fewer fresh token's worth of learning.
2. Memorisation. A document seen 100 times is far more likely to be recited verbatim at inference, which is both a privacy and a robustness liability.
3. Distribution skew. Frequent duplicates over-represent their content; the model becomes biased toward whatever boilerplate dominates the duplication.

6.2 MinHash + LSH

The standard near-deduplication algorithm. For each document $d$:

1. Extract n-gram shingles (typically 5-grams of words). Call this set $S(d)$.
2. Compute $r$ MinHash values: for each of $r$ independent random hash functions $h_i$, record $\min_{s \in S(d)} h_i(s)$. The resulting $r$-vector is the MinHash sketch.
3. The Jaccard similarity $\frac{|S(d_1) \cap S(d_2)|}{|S(d_1) \cup S(d_2)|}$ is estimated by the fraction of matching components in the two sketches.

Then Locality-Sensitive Hashing (LSH) partitions the $r$ MinHash components into bands and buckets documents that share any band exactly. Documents in the same bucket are candidate near-duplicates; compute their Jaccard and drop those above threshold.

For a Common Crawl-scale corpus, this scales to billions of documents in $\sim 100$ CPU-days. Lee et al. (2022)[8] found that:

• 5-gram shingles work well for English.
• A Jaccard threshold of $0.8$ catches most near-duplicates without false-positives on summarised content.
• After deduplication, models converge $\sim 1.5\times$ faster and have lower verbatim memorisation.

6.3 Practical pipelines

The open-source text-dedup library (HuggingFace) implements MinHash + LSH at scale. DCLM and FineWeb use it as a subprocess. The deduplication budget is typically:

• Within-snapshot dedup: deduplicate within one Common Crawl snapshot (~100 B tokens, hours to a day on a large cluster).
• Cross-snapshot dedup: deduplicate across all years of CC used (much heavier; sometimes skipped, sometimes done block-wise).
• Cross-source dedup: deduplicate the web crawl against Wikipedia, books, etc. (catches verbatim mirrors of curated sources in the web).

A pretraining pipeline that skips deduplication can still produce a working model — early GPT-2 essentially did — but the loss is measurably higher per token of training. Modern pipelines treat dedup as mandatory.

7. What the model "knows" after pretraining

A model trained on the corpora above acquires three rough kinds of knowledge:

1. Linguistic competence. Syntax, morphology, register, idiom. Available even from poor-quality data.
2. Factual / world knowledge. Wikipedia, news, encyclopedic web pages. Sensitive to data quality and recency.
3. Reasoning patterns. Math derivations (arXiv, math forums), code logic (GitHub), step-by-step explanations (Stack Exchange, tutorial blogs, educational web). FineWeb-Edu's emphasis on educational content is specifically aimed at this third category.

Alignment (Cap 9) builds on top of these capabilities; it does not create them. A model that doesn't see code in pretraining cannot be made to write code by instruction-tuning alone. The pretraining mixture decides the capability ceiling; alignment shapes how the model uses what it already has.[14]

8. What we keep going forward

Three numbers and three concepts:

• Pipeline: extract → language ID → heuristic filter → learned filter → deduplicate → decontaminate → (optional) PII.
• Deduplication is the highest-leverage step. MinHash + LSH at Jaccard $\geq 0.8$, 5-gram shingles.
• Quality beats quantity. Classifier-filtered, deduplicated 4 T tokens beats raw 10 T tokens at fixed compute.

Next session is the other side of the data problem — tokenization: how raw UTF-8 text becomes the integer sequences the Transformer consumes.