Pretraining Pipeline

TL;DR — Pretraining is a data pipeline (extract → language-ID → quality filter → deduplicate → decontaminate → tokenise → pack) wrapped around a distributed training loop. Each stage trades corpus size for corpus quality, and the trade is empirical: on the 2013-48 Common Crawl snapshot, global deduplication removed 90% of the documents and made the survivors worse (Penedo et al. 2024, arXiv:2406.17557). Your compute budget $C$ is one scalar, and the Chinchilla fit $\hat{L}(N,D)=E+A/N^{\alpha }+B/D^{\beta }$ says how to split it between parameters $N$ and tokens $D$: roughly $D\approx 20N$. Modern frontier models break that ratio deliberately — Llama 3 trained on 15T tokens — because inference, not training, dominates lifetime cost.

1. Why this exists

A base model is a lossy compression of a corpus. Every decision about the corpus is a decision about the model, and none of it is visible in the architecture diagram.

Concrete failure #1 — memorisation and contaminated evals. Lee et al. 2022 (arXiv:2107.06499) found that over 1% of unprompted model output is copied verbatim from the training set, that a single 61-word English sentence appears more than 60,000 times in C4, and that train–test overlap affects over 4% of the validation sets of standard datasets. Deduplicating C4 (shrinking it by up to 19%) cut memorised emissions by 10× with no perplexity cost. That is a pipeline bug, not a modelling one.

Concrete failure #2 — mis-spending the compute budget. Compute is usually fixed in advance (“2,048 GPUs for three weeks”). Following Kaplan et al. 2020 (arXiv:2001.08361), the field spent it by growing models at roughly fixed data: Gopher (280B), GPT-3 (175B) and MT-NLG (530B) all saw ~300B tokens. Hoffmann et al. 2022 (arXiv:2203.15556) re-ran Gopher’s budget and found the optimum was 4× smaller and 4× longer-trained — 70B parameters on 1.4T tokens — and that Chinchilla beat Gopher, GPT-3 and MT-NLG on essentially every benchmark. Same money, better model, opposite allocation.

Where this matters for you. An interview question “how are LLMs trained?” is not about backprop. It is about the pipeline: what gets thrown away and why, what a token is, and how the budget is split.

2. How it works

2.1 The pipeline end to end

flowchart LR
    CC["Common Crawl WARC, 96 monthly snapshots, PB scale"] --> EX["Text extraction (trafilatura)"]
    EX --> LID["Language ID (fastText, p >= 0.65)"]
    LID --> HQ["Heuristic quality filters"]
    HQ --> MB["Model-based filter (perplexity or classifier)"]
    MB --> DD["Deduplication (MinHash + LSH)"]
    DD --> DC["Decontamination vs eval sets"]
    DC --> TK["Tokenizer training and tokenisation"]
    TK --> MX["Mixture with books, code, papers, Wikipedia"]
    MX --> PK["Packing into fixed-length sequences"]
    PK --> TR["Distributed training loop, 6ND FLOPs"]

Everything left of TK is CPU- and I/O-bound; everything right is GPU-bound. That is why the data pipeline, not the training run, is usually the schedule risk.

2.2 Where the data comes from

The bulk is Common Crawl, a non-profit publishing monthly web snapshots since 2007; FineWeb consumed 96 of them. It ships WARC (raw HTML plus metadata) and WET (a server-side text extraction). FineWeb’s first ablation compared WARC-with-trafilatura against WET: trafilatura won clearly, because WET retains navigation and boilerplate. That single choice beat most filter tuning, and it is expensive CPU.

The rest of a real mixture is deliberately non-web: books, code, papers, Wikipedia, synthetic data. GPT-3’s mixture is in Brown et al. 2020 (arXiv:2005.14165); Gopher’s MassiveText is 2.35B documents / ~10.5TB sampled non-uniformly (Rae et al. 2021, arXiv:2112.11446). Mixture weights decide what the model is good at, and they are usually changed for a short annealing phase at the end — Llama 3 upsamples high-quality sources there and extends context 8K→128K.

2.3 Heuristic filters — and what they actually remove

The canonical set is Google’s Gopher/MassiveText rules (Rae et al. 2021, Appendix A), reused by RefinedWeb and FineWeb:

The numbers matter more than the list. FineWeb measured each candidate by training a 1.71B model on ~28B tokens and evaluating downstream:

• C4’s terminal-punctuation filter gave the single largest boost — and removed ~30% of all tokens. FineWeb dropped it for that reason.
• FineWeb’s three winning custom filters removed 10.14% (fraction of lines ending in punctuation $\le 0.12$), 12.47% (fraction of characters in duplicated lines $\ge 0.1$) and 3.73% (fraction of lines shorter than 30 chars $\ge 0.67$). The rule sets overlap (a document can be removed by more than one), so these percentages do not simply add; the ~22% together figure is quoted as reported by the FineWeb source, for about +1% aggregate score. The retained C4 rules each remove < 5%.

A good filter removes a lot of bad data and a little good data. You find the threshold by overlapping the histograms of a high-quality and a low-quality slice — not by intuition.

2.4 Model-based filtering

Perplexity filtering. CCNet (Wenzek et al. 2019, arXiv:1911.00359) trains a 5-gram Kneser-Ney model with KenLM on Wikipedia, scores every paragraph, and keeps those below a tercile-derived perplexity threshold. It is fast and language-portable, but it bakes in the reference corpus, so good dialogue gets deleted.

Classifier filtering. FineWeb-Edu (arXiv:2406.17557) had Llama-3-70B-Instruct score 460,000 pages for educational quality on a 0–5 additive scale, then fit a linear regressor on frozen Snowflake-arctic-embed-m embeddings (410k/50k split, 20 epochs, lr 3e-4), hitting 82% F1 at threshold 3. The 1.3T-token subset lifted MMLU 33%→37% and ARC 46%→57% at 1.71B parameters, at a cost of 6,000 H100 GPU-hours to score 15T tokens. Caveat: it down-samples business, law, entertainment and travel because it encodes the annotator’s taste.

2.5 Deduplication, precisely

Near-duplicate detection at web scale needs MinHash to compress a document into a small signature, and locality-sensitive hashing (LSH) to find candidate pairs without an $O(n^2)$ join.

Steps 1–2 — shingles and MinHash. Reduce each document to its set of $k$-grams (FineWeb: 5-grams over an English word tokenizer; too small and unrelated English documents look alike, too large and near-duplicates fall below threshold), then, for shingle sets $A,B$ and independent hashes $h_1,\dots ,h_k$:

$$\mathrm{sig}(A)=(\underset{a\in A}{\min }{h}_1(a),\dots ,\underset{a\in A}{\min }{h}_k(a)),\hat{J}(A,B)=\frac{1}{k}\sum _{i=1}^{k}1\left[h_i(A)=h_i(B)\right]$$

• $A,B$ — the two documents’ shingle sets; the object whose similarity is being estimated.
• $h_i$ — the $i$-th hash. It makes the minimum random: under a random permutation, the probability the smallest element of $A\cup B$ lies in $A\cap B$ is exactly $|A\cap B|/|A\cup B|$, so MinHash samples that event without materialising a permutation.
• ${\min }_{a\in A}{h}_i(a)$ — one integer per hash, so the signature is $k$ integers rather than the document.
• $k$ (num_perm) — signature length and the only variance knob: $\mathrm{Var}(\hat{J})=J(1-J)/k$. At $J=0.5$, $k=128$ gives ±4.4 points, $k=16$ gives ±12.5% and starts merging unrelated documents, $k=512$ costs memory for ±2.2%.
• $1[\cdot ]$ — the indicator, 1 when the minima agree; averaging indicators turns set similarity into an unbiased coin-flip estimate.

Step 3 — LSH banding. Split the $k$ hashes into $b$ bands of $r$ rows ($k=br$); two documents are candidates if any band matches exactly:

$$P(\text{candidate}\mid J)=1-{\left(1-J^r\right)}^b$$

• $J$ — true Jaccard similarity.
• $r$ — rows per band. $J^r$ is the probability all $r$ independent MinHash events agree in one band, so raising $r$ makes a band match much harder.
• $1-J^r$ — the probability this band disagrees; $b$ — the number of independent bands, so $(1-J^r{)}^b$ is the probability every band disagrees and the whole expression is the probability that at least one matches.
• The S-curve’s knee sits near $J^{\star }\approx (1/b{)}^{1/r}$. FineWeb’s 112 hashes in 14 bands of 8 give $(1/14{)}^{1/8}\approx 0.72$, exactly its stated 75% target — so the parameters are derivable: fix the threshold, choose $(b,r)$ for the knee, then set $k=br$ for the variance you can afford.
• Remove $r$ ($r=1$): $P\approx 1-(1-J{)}^b$, almost everything is a candidate and the index degenerates into a quadratic join. Remove $b$ ($b=1$): $P=J^r=0.75^8=0.10$ — you silently miss 90% of true duplicates.

Step 4 — cluster and keep one. Similar documents cluster transitively (A~C and B~C groups A, B, C); one representative survives per cluster.

How to deduplicate is counterintuitive. FineWeb’s first attempt deduplicated globally across all 96 snapshots, newest first. The oldest snapshots, already deduplicated against everything newer, lost up to 90% of their documents — and training on survivors versus removed documents showed the removed set was better. Per-snapshot dedup is what improved benchmarks. Hypothesis: the win is killing giant clusters (hundreds of thousands of copies); removing small clusters (< ~100 copies) discards legitimately rare text.

2.6 Tokenisation

The tokeniser is trained on your corpus and then frozen. It decides how much text fits in a context window and how compute is spent.

BPE (Sennrich et al. 2015, arXiv:1508.07909, imported from compression): start from the 256 byte values, repeatedly merge the most frequent adjacent symbol pair into a new vocabulary entry, until $V$ is reached, so $V=256+\text{merges}$. The 256-byte alphabet is what guarantees no out-of-vocabulary token ever — any UTF-8 string is encodable, at worst one byte per token. Drop it and unseen scripts become UNK.

SentencePiece (Kudo & Richardson 2018, arXiv:1808.06226) operates on the raw character stream with no whitespace pre-tokenisation, treating space as an ordinary character (▁). That makes it language-agnostic, which matters for Chinese, Japanese and Thai. It implements BPE and a unigram language model: start from a large seed vocabulary, prune by likelihood, score segmentations probabilistically. The probabilistic view buys subword regularization — a different segmentation each epoch, which acts as data augmentation.

tiktoken is OpenAI’s Rust BPE. cl100k_base has 100,256 base merge entries (GPT-3.5/GPT-4 era); o200k_base has 199,998 (GPT-4o era); plus special tokens (openai/tiktoken) [as of 2026-10]. Llama 3 hybridised: 100K tiktoken tokens + 28K non-English tokens = a 128,000-token vocabulary.

Vocabulary size is a real parameter. With untied input/output embeddings (LLaMA-family), embeddings cost $2\cdot V\cdot d_{model}$: Llama 2 ($V=32,000$, $d=4096$) ≈ $2.6\times 10^8$ parameters; Llama 3 ($V=128,000$) ≈ $1.05\times 10^9$ — 3% versus 13% of an 8B model. Those are also the parameters with the thinnest gradient (one row per token, most rows rare). The payoff is compression: Llama 3’s English rate rose 3.17 → 3.94 chars/token, so the same token budget covers ~24% more text.

The fairness cost. Fertility (tokens per word) is language-dependent. Petrov et al. 2023 (arXiv:2305.15425) measure the disparity — extra tokens a non-English string costs relative to a meaning-equivalent English string — and find large gaps for English-heavy tokenisers. A Hindi user pays several times the tokens, context and money for the same content. Adding 28K non-English tokens is the direct mitigation.

2.7 The objective, and why 6ND

With token sequence $x_1,\dots ,x_T$:

$$L(\theta )=-\frac{1}{T}\sum _{t=1}^T\log p_{\theta }\left(x_t\mid x_{<t}\right)$$

• $x_t$ — the target token at position $t$. It is the entire label.
• $x_{<t}$ — the context: every earlier token, enforced by the causal mask (Attention §2.4). It is load-bearing: without it the model reads the answer, the loss collapses, nothing generalisable is learned.
• $p_{\theta }(\cdot \mid x_{<t})$ — the next-token distribution: softmax over the vocabulary applied to the logits at position $t-1$. Note the one-position shift; logits at $t$ predict $x_{t+1}$. Getting this wrong is the classic silent bug — the model learns the identity map, loss → 0, generation is garbage.
• $\log$ — natural log, so the loss is in nats. ${\log }_2$ reports bits and rescales every gradient by a constant; the optimum is unchanged, a tuned LR is not.
• $-\frac{1}{T}\sum$ — empirical mean NLL. The minus turns a likelihood into a loss ($\log p\le 0$, so $L\ge 0$). The $1/T$ is not cosmetic: without it loss and gradient scale with sequence length, so doubling $T$ equals doubling the LR.
• $\exp (L)$ is perplexity — a monotone reparameterisation of the same number.

Training FLOPs for a dense transformer:

$$C\approx 6ND$$

• $N$ — non-embedding parameters. Embeddings are lookups, not matmuls, so they cost $O(1)$ per token, not $2N$.
• $D$ — training tokens, each one forward + backward pass.
• $2ND$ — forward: per token and per weight, one multiply and one add.
• $4ND$ — backward, conventionally twice the forward pass. Total $\approx 6ND$. It ignores attention’s $O(Dnd)$ term, but Hoffmann et al. 2022 Appendix F checked it per-layer and found the ratio close to 6 across realistic configs. Use it for capacity planning. Using $2ND$ (forward only) underestimates a run by 3×.

2.8 Scaling laws: Chinchilla, term by term

Hoffmann et al. 2022 fit > 400 models (70M–16B parameters; 5B–500B tokens) to

$$\hat{L}(N,D)=E+\frac{A}{N^{\alpha }}+\frac{B}{D^{\beta }}$$

• $\hat{L}(N,D)$ — predicted final loss in nats as a function of size and data.
• $E$ — the irreducible loss, the entropy of natural text: the floor nothing beats, because language is genuinely unpredictable next-token. Remove it and the fit is forced through the origin, predicting loss 0 as $N,D\to \infty$.
• $A/N^{\alpha }$ — the finite-capacity term: an $N$-parameter transformer cannot represent the true distribution. Remove it and loss stops depending on size — the reason to scale at all disappears.
• $B/D^{\beta }$ — the finite-data term: the model has not converged because it saw only $D$ tokens. Remove it and loss stops depending on data, i.e. the fit claims a fixed dataset never saturates.
• $\alpha$ — decay exponent of the capacity term, fitted $0.34$; $\beta$ — decay exponent of the data term, fitted $0.28$. Larger exponent = that axis pays off faster. Fitted constants: $E=1.69$, $A=406.4$, $B=410.7$ (Appendix D.2, Eq. 10). Note $A\approx B$ but $\alpha >\beta$: at this fit parameters were marginally the better buy — the opposite of the naive reading of “Chinchilla says data matters more”.
• The form is justified, not arbitrary: a classical risk decomposition into Bayes risk ($E$), functional approximation ($\propto 1/N^{1/2}$, Siegel & Xu 2020) and stochastic approximation/early stopping ($\propto 1/D^{1/2}$, Robbins & Monro 1951). The fitted exponents land below $1/2$, which the paper flags as the key empirical finding.

The optimisation. Loss decreases monotonically in both $N$ and $D$, so unconstrained the minimum is at infinity; the constraint is what makes the problem well-posed:

$$C\approx 6ND⟹D=\frac{C}{6N}$$

$$\hat{L}(N)=E+A{N}^{-\alpha }+B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta },\frac{d\hat{L}}{dN}=-\alpha A{N}^{-\alpha -1}+\beta B{\left(\frac{6}{C}\right)}^{\beta }{N}^{\beta -1}=0$$

Setting the derivative to zero and collecting $N$ yields the closed form of Eq. 4:

$$N_{opt}(C)=G{\left(\frac{C}{6}\right)}^a,D_{opt}(C)=G^{-1}{\left(\frac{C}{6}\right)}^b$$

$$G={\left(\frac{\alpha A}{\beta B}\right)}^{\frac{1}{\alpha +\beta }},a=\frac{\beta }{\alpha +\beta },b=\frac{\alpha }{\alpha +\beta }$$

• $G$ — the leading constant bundling the fitted coefficients; it sets the level of the frontier. With the fitted values, $G\approx 1.34$.
• $a=\beta /(\alpha +\beta )=0.28/0.62\approx 0.45$ — how fast parameters grow with compute; $b=\alpha /(\alpha +\beta )=0.34/0.62\approx 0.55$ — how fast data grows. By construction $a+b=1$.
• The paper reports $a=0.50,b=0.50$ (approach 1, training-curve envelope), $0.49/0.51$ (approach 2, IsoFLOP profiles) and $0.46/0.54$ (approach 3, the parametric fit above). All three say “scale them equally” — the headline is robust to the fitting method. Kaplan et al. reported $0.73/0.27$; the diagnosis is that they used one fixed cosine-schedule length for every run, so shorter runs were scored before decay finished, inflating their loss and making data look wasteful.

The practical rule. Approach 1’s Table 3 points lie on a 20-tokens-per-parameter line — 400M → 8.0B, 1B → 20.2B, 10B → 205.1B, 67B → 1.5T — which as a closed form is

$$C\approx 6N_{opt}{D}_{opt},D_{opt}=20N_{opt}⟹N_{opt}=\sqrt{\frac{C}{120}}=\frac{1}{\sqrt{20}}{\left(\frac{C}{6}\right)}^{1/2}\approx 0.2236{\left(\frac{C}{6}\right)}^{1/2}$$

$$D_{opt}=20N_{opt}=\sqrt{20}{\left(\frac{C}{6}\right)}^{1/2}\approx 4.4721{\left(\frac{C}{6}\right)}^{1/2}$$

and reproduces the table to within rounding (at $C=6\cdot 10^9\cdot 20.2\times 10^9\approx 1.21\times 10^{20}$: the fit gives $N_{opt}\approx 1.0$B and $D_{opt}\approx 20.1$B, against the table’s $1$B / $20.2$B). Chinchilla itself is 70B on 1.4T tokens — the same line.

Why modern models are far off it. $D\approx 20N$ minimises loss per training FLOP, not lifetime cost. A production model pays for inference forever, so overspending on training to get a smaller model is rational: Llama 3 trained on 15T tokens (~1,900 tokens/parameter for the 8B model), DeepSeek-V3 on 14.8T (arXiv:2412.19437).

3. Implementation

3.1 A runnable near-duplicate detector

datasketch==2.0.0 [as of 2026-10] changed the default permutation scheme to "affine32" (fixes a similarity over-estimation bias, halves sketch memory); hashes are not compatible with 1.x, so persisted indexes must be rebuilt.

# pip install "datasketch==2.0.0"
import re
from datasketch import MinHash, MinHashLSH

NUM_PERM = 128      # k: std err sqrt(J(1-J)/k) ~= 0.044 at J=0.5
THRESHOLD = 0.8     # LSH places its S-curve knee internally for this threshold
WORD = re.compile(r"\w+")

def minhash(text: str, k: int = 5) -> MinHash:
    """Word-level 5-shingles, as FineWeb does over an English word tokenizer."""
    words = WORD.findall(text.lower())
    m = MinHash(num_perm=NUM_PERM)
    for i in range(max(0, len(words) - k + 1)):
        m.update(" ".join(words[i:i + k]).encode("utf-8"))
    return m

def dedupe(docs: list[str]) -> list[str]:
    lsh = MinHashLSH(threshold=THRESHOLD, num_perm=NUM_PERM)
    kept = []
    for i, doc in enumerate(docs):
        m = minhash(doc)
        if lsh.query(m):          # candidates; Jaccard confirms internally
            continue
        lsh.insert(f"doc-{i}", m)
        kept.append(doc)
    print(f"kept {len(kept)}/{len(docs)}")
    return kept

Two things decide whether this survives production:

• num_perm must match the index. Mixing 128-perm queries against a 64-perm index returns wrong candidates, silently.
• The threshold is an LSH property. MinHashLSH(threshold=t, num_perm=k) derives its own banding; the §2.5 derivation is how you check whether the resulting S-curve is steep enough. Run per snapshot, not globally (§5).

At web scale you would not hold docs in memory; the real pipeline keeps per-shard signatures on disk and does a distributed band-and-merge. The FineWeb codebase is datatrove (datatrove==0.10.1 [as of 2026-10]; see examples/fineweb.py).

3.2 The cross-entropy loss, written correctly

import torch.nn.functional as F

def next_token_loss(logits, input_ids, ignore_index=-100):
    """logits: (B, T, V). input_ids: (B, T). Mean NLL over scored tokens."""
    shift_logits = logits[:, :-1, :].contiguous()   # predict t+1 from position t
    shift_labels = input_ids[:, 1:].contiguous()
    return F.cross_entropy(
        shift_logits.view(-1, shift_logits.size(-1)),
        shift_labels.view(-1),
        ignore_index=ignore_index,   # divides by the count of NON-ignored targets
    )

• F.cross_entropy with ignore_index divides by the number of non-ignored targets, which is exactly the denominator you want for SFT masking.
• Remove the [:, :-1] / [:, 1:] shift and the loss drops near zero within hundreds of steps while generation stays random.

3.3 The knobs that matter

4. Numbers and trade-offs

4.1 What a training run costs

$$\text{GPU-hours}=\frac{6ND}{\text{MFU}\times \text{peak FLOP/s}\times 3600}$$

• MFU — model FLOPs utilisation, the fraction of peak you actually reach. 0.35–0.55 is normal for a well-tuned large run; under 0.2 for a badly configured one.
• peak FLOP/s — an H100 SXM quotes 1,979 TFLOPS BF16 with sparsity, i.e. 989 TFLOPS dense (NVIDIA H100 datasheet: “shown with sparsity. Specifications 1/2 lower without sparsity”). Use the dense number.
• Worked (example): $N=8\times 10^9$, $D=15\times 10^{12}$ → $C=6\cdot 8\times 10^9\cdot 15\times 10^{12}=7.2\times 10^{23}$ FLOPs. At MFU 0.40 on H100s: $7.2\times 10^{23}/(0.4\cdot 9.89\times 10^{14})=1.82\times 10^9$ s $=5.05\times 10^5$ GPU-hours ≈ 505,000 H100-hours. For scale, Llama 3 405B ran on up to 16K H100s at $3.8\times 10^{25}$ FLOPs — ~$2.7\times 10^7$ GPU-hours at the same MFU (example), i.e. 53× the 8B/15T run.

MFU is where the engineering lives: tensor parallelism within a layer, pipeline parallelism across nodes, FSDP/ZeRO to shard optimizer state, activation checkpointing to trade compute for memory, and a loader that never stalls the GPU.

4.2 Memory

Mixed-precision Adam needs ~16 bytes/parameter: 2 (bf16 weights) + 2 (bf16 grads) + 4 (fp32 master) + 4 (Adam $m$) + 4 (Adam $v$). A 7B model is ~112GB of state before activations — more than one 80GB H100. ZeRO (Rajbhandari et al. 2020, arXiv:1910.02054) and FSDP partition optimizer state, gradients and optionally parameters across data-parallel ranks.

4.3 Data size

15T tokens is tens of terabytes of text. FineWeb’s funnel: base filtering of trafilatura-extracted WARC left ~36T tokens (GPT-2 tokeniser); per-snapshot MinHash took it to ~20T; C4 plus custom filters left 15T. Roughly 60% of base-filtered web text never reaches the final dataset; the ~40% that survives is the product.

4.4 Which axis you are optimising

Every choice above moves one axis: filtering and dedup buy loss per token at the cost of tokens remaining; vocabulary and token count buy serving cost at the cost of training compute.

5. Pitfalls

Evaluating on contaminated data. Symptom: implausibly high benchmark scores that fall 5–20 points on a genuinely held-out set. Over 4% of standard validation sets overlap training text (Lee et al. 2022). Decontaminate before quoting numbers.

Global dedup across snapshots. Symptom: the deduplicated dataset scores worse than raw, and the oldest snapshots lost ~90% of documents. Fix: per-snapshot dedup unless measured otherwise.

Filtering the corpus too small. Symptom: training loss looks healthy but downstream benchmarks plateau early. Filtering trades tokens for quality at an exchange rate only an ablation can measure.

Packing without resetting positions or masking boundaries. Symptom: mid- sentence topic jumps; answers referencing a “previous conversation” that is actually the next packed example. Packing needs a block-diagonal mask, or a varlen kernel plus per-document position_ids and cumulative sequence lengths. TRL emits exactly position_ids, cu_seq_lens_q/k, max_length_q/k, seq_idx.

Truncating documents at a fixed length. Symptom: the model hallucinates about code (a variable definition was cut), summarises unfaithfully, or “doesn’t know” a fact that was in its corpus. Ding et al. 2024 (arXiv:2404.10830) call this the curse of truncation; best-fit packing truncates only when no legal grouping exists and improves performance, including +16.8% on a context-following measure.

Comparing perplexity across tokenisers. Symptom: an obviously better model reports worse perplexity after a tokeniser change. Report bits per byte (arXiv:2101.00027); Llama 2 covered 3.17 chars/token where Llama 3 covered 3.94.

Off-by-one in the loss shift. Symptom: training loss near zero within a few hundred steps, samples still random. The model learned to copy its input.

Reusing data for too many epochs. Symptom: the loss curve flattens and more data stops helping. Muennighoff et al. 2023 (arXiv:2305.16264) trained > 400 models up to 1,500 epochs: up to 4 epochs of repeats is essentially free, returns decay after, and by 40 epochs repeating is worthless.

6. Using it in a project

The default is: do not pretrain. You will not beat a released base checkpoint on general text, and the run above costs half a million H100-hours before the first data filter. Order the decision:

Continued (domain-adaptive) pretraining is the realistic middle ground, and the one place pretraining belongs in an application project. Gururangan et al. 2020 (arXiv:2004.10964) showed that continuing on in-domain unlabelled text helps — and that their recipe keeps a general-domain mix, because narrow continued pretraining causes catastrophic forgetting. Concretely:

• Budget $10^9$–$10^{10}$ tokens, not $10^{12}$.
• Keep 50–80% general text by token count, or you lose instruction following, code and general benchmarks.
• Reuse the model’s own tokeniser and sequence length; changing either invalidates the pretrained embeddings.
• Treat general benchmarks (MMLU, HellaSwag) as regression tests with equal seriousness to the domain metric.

When pretraining really is right: the domain is far from web text (proteins, logs, a new language), the released tokeniser mangles it, and you have the corpus and GPUs for at least a small model. Even then, publish the pipeline ablation, not the model.

7. How you would prove it worked

Copy FineWeb’s design: hold the model and the token count fixed, vary exactly one stage, train two seeds, and compare downstream benchmarks plus loss in bits per byte.

• Budget: 1.71B parameters, ~28B tokens for filter ablations (roughly Chinchilla-optimal for that size) and 350B when the effect is below the noise at 28B. Same architecture, same LR schedule, same token count per arm.
• Arms: baseline (previous stage only) plus one change. Never two.
• Metrics: low-variance small benchmarks at 1.7B — CommonSenseQA, HellaSwag, OpenBookQA, PIQA, SIQA, WinoGrande, ARC, MMLU, truncated to 1,000 samples — plus validation bits-per-byte.
• Cost: FineWeb trained > 70 models for ~80,000 H100-hours. That is what a defensible data decision costs.

The comparisons that decide things: filter hard/fewer tokens versus filter light/more tokens at equal tokens (the answer flips between 28B and 350B); per-snapshot vs global vs no dedup; and an n-gram contamination audit of your eval sets against the final corpus.

Reporting template (example — replace with your own numbers). “We ablated the pipeline in five stages at a fixed 28B-token budget on a 1.71B model, two seeds per arm. Adding the retained C4 filters plus three custom heuristic filters removed 22% of tokens and raised the 8-task aggregate ~1.0 point (example). A global-dedup variant removed 90% of the oldest snapshot and lost 0.4 points (example), so we kept per-snapshot dedup. Eval contamination after decontamination was 0.3% of items (example), down from 4.1% (example). Total ablation cost: 80,000 H100-hours (example).”

8. Interview drill

“Where does pretraining data come from, and what do the filters remove?” Strong answer: mostly Common Crawl snapshots, extracted from WARC with trafilatura (not WET, which keeps boilerplate), plus books, code, papers and Wikipedia. Filters are heuristic (word count, stop-word ratio, symbol ratio, n-gram/line/paragraph repetition) and model-based (CCNet’s KenLM perplexity against Wikipedia; FineWeb-Edu’s embedding + linear-regression classifier). They remove a lot: C4’s terminal-punctuation rule alone is ~30% of tokens; FineWeb’s three custom filters are ~22% together as reported (their rule sets overlap, so the individual percentages do not simply add). Common wrong answer: “crawl the web and tokenise it.” That cannot explain why models trained on the same crawl differ by several benchmark points.

“Why 6ND, and how do you turn it into GPU-hours?” Strong answer: 2ND forward (multiply + add per weight per token) plus 4ND backward (2× forward), so 6ND, counting non-embedding parameters because embedding lookups are not matmuls. Divide by MFU × dense peak FLOPs × 3600. Common wrong answer: quoting 2ND (forward only) and being 3× optimistic, or using the 1,979 TFLOPS sparsity figure on an H100 instead of 989 dense.

“Why did Chinchilla disagree with Kaplan et al.?” Strong answer: Kaplan et al. used one fixed cosine-schedule length for all runs, so shorter runs were scored pre-decay, inflating their loss and making data look wasteful. Chinchilla matched schedule length to token budget and added larger models, finding negative curvature in the FLOP–loss frontier — giving $a\approx b\approx 0.5$ across three fitting methods instead of 0.73/0.27. Common wrong answer: “Chinchilla says 20 tokens per parameter.” That is the practical corollary; the result is “scale them equally.”

“Why do modern models use thousands of tokens per parameter?” Strong answer: Chinchilla minimises loss per training FLOP; production pays inference forever, so overspending on training to get fewer parameters is rational. That is the inference-aware frontier. Common wrong answer: “the scaling laws were wrong.” They were not; the objective changed.

“Explain MinHash and LSH. What do the parameters control?” Strong answer: MinHash estimates Jaccard similarity as the fraction of $k$ independent hashes whose minimum agrees, with variance $J(1-J)/k$. LSH bands the $k$ hashes into $b$ bands of $r$ rows and calls a pair a candidate on any band match: $P=1-(1-J^r{)}^b$, an S-curve with knee near $(1/b{)}^{1/r}$. FineWeb’s 14 bands of 8 put the knee at ~0.72, matching its stated 75% target. Common wrong answer: “hashing to find exact duplicates” — MinHash is fuzzy; exact matching is a separate Bloom-filter pass.

“What is the most expensive part of the pipeline?” Strong answer: it splits. The training loop is the GPU cost ($6ND$/MFU, ~$5\times 10^5$ H100-hours for 8B/15T); WARC extraction, MinHash and LSH are CPU/IO-bound over petabytes and dominate wall-clock; a learned filter is its own GPU job — scoring 15T tokens with the FineWeb-Edu classifier took 6,000 H100-hours.

Next: Supervised Fine-Tuning (SFT) — a base model is a text completer; SFT is what turns it into something that answers.