Part 02 — Pretraining and Fine-Tuning
How it is trained.
Part 01 described the machine; this part describes how it acquires behaviour. The organising idea is that there are three different ways to change what a model does, and they are ordered by cost: change the prompt (free), adapt a small number of parameters (cheap), or retrain (expensive). Most engineering mistakes in this area are choosing the expensive option for a problem the cheap one solves.
- Pretraining Pipeline — the data pipeline (extract, filter, dedupe, tokenise, pack), the empirical quality/size trade behind it, and the Chinchilla fit that splits compute between parameters and tokens.
- Supervised Fine-Tuning (SFT) — turning a text completer into an instruction follower: the chat template, masking the prompt out of the loss, and the superficial-alignment hypothesis. SFT changes format, not knowledge, so 1,000 examples (LIMA) go a long way.
- LoRA and QLoRA — parameter-efficient fine-tuning. How a low-rank update works, what the rank and alpha actually control, how much memory it saves, and the learning-rate trap that makes it look broken.
- Quantization — PTQ vs QAT, why outlier activations make LLM quantization hard, the format zoo (INT8/INT4/NF4/GPTQ/AWQ/GGUF), and how to measure the accuracy you gave up.
Depends on: Part 01 for the machine that is being trained.
Where this leads: Part 03 picks up where SFT stops, and is not readable without the LoRA/quantization vocabulary from here.