Part 01 — Foundations

How the model is built.

The goal of this part is narrow and mechanical: by the end you should be able to draw the inside of a transformer block from memory and justify every piece of it. Nothing here is trivia — every later chapter is a consequence of these mechanisms. LoRA exists because of how the weight matrices are shaped. The KV cache exists because of the attention equation. Every context-extension trick exists because of how RoPE rotates positions.

Read the chapters in order; each assumes the previous one.

1. The Transformer — the end-to-end block: attention, feed-forward, residual stream, layer norm, and why the residual stream is the right mental model for everything that follows.
2. Attention — the core mechanism, derived rather than asserted. This is the reference chapter for the whole book: it shows the depth every other chapter is written to.
3. Positional Encoding — attention is permutation-invariant, so it cannot tell “dog bites man” from “man bites dog”. Sinusoidal, learned, ALiBi and RoPE, and why RoPE won.
4. Modern Architecture Deltas — the five substitutions between the 2017 paper and a 2026 model: pre-norm, RMSNorm, SwiGLU, GQA, and FlashAttention. Each gets the problem it solved at scale and the cost it added. (MoE appears in the interview drill, not as a full delta.)
5. BERT vs GPT vs Encoder-Decoder — the three architectures, their training objectives, and why decoder-only won for general-purpose models while encoders still own retrieval and classification.

Depends on: nothing; this is where the book starts.

Where this leads: Part 02 takes these mechanisms as given and asks how the model is trained. Nothing later is readable without the residual stream and the attention shape from here.