Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Part 03 — Alignment and Reinforcement Learning

How it is made to behave.

This is the densest part of the book, and the one most often answered badly in interviews, because most candidates can name the algorithms but cannot say what problem each one removes. The through-line is a sequence of cost reductions:

  • PPO works, but keeps four models in memory at once — actor, critic, reward model, and reference.
  • GRPO removes the value model by replacing the learned baseline with the mean reward of a group of samples.
  • DPO removes the reinforcement loop entirely by turning the preference objective into a classification loss on pairs.

Read in order. The mathematics of GRPO and DPO is not intelligible without the policy/value vocabulary built in the first three chapters.

  1. RLHF: The Three Stages — why next-token prediction is the wrong objective for helpfulness, and how SFT → reward model → RL is assembled.
  2. Reward Modeling — the Bradley–Terry preference model, why a reward model is a proxy that can be gamed, and what reward hacking looks like in a training curve.
  3. PPO for LLMs — the clipped surrogate objective derived term by term, the KL penalty and why a reference model is mandatory, and the four-models-in-memory problem.
  4. GRPO — group-relative advantage estimation, exactly what removes the critic, what it costs, and why it unlocked reasoning models.
  5. DPO — the derivation that collapses RLHF into a pairwise loss, its implicit reward, and the conditions under which it underperforms PPO.
  6. Monte Carlo, TD, and Search — the estimator family underneath all of the above: Monte Carlo returns vs bootstrapping, the bias–variance trade-off, and how MCTS with PUCT turns a value estimate plus a policy prior into a search.
  7. RL Frameworks — the systems half: how the four roles are colocated and scheduled across GPUs, how weights move from trainer to rollout engine, and what OpenRLHF, veRL, SkyRL and DeepSpeed-Chat each abstract. TRL no longer ships a PPO trainer.

After this part you can answer: why PPO needs a value function and GRPO does not; what the KL term is protecting against; why DPO can be written as a classification loss; and what UCB is doing in a tree search.

Depends on: Part 02 for the SFT model that RL starts from. The policy/value vocabulary this part needs is built in its own first three chapters, not earlier.

Where this leads: Part 04 takes the aligned model and asks what it costs to serve.