RLHF: The Three Stages

TL;DR — Next-token likelihood is the right loss for modelling text and the wrong loss for behaving well: it scores token frequencies, not outcomes, and it has no way to express “either of these two answers is fine, but this third one is not”. RLHF replaces the likelihood objective with a scalar reward learned from human comparisons, and optimises the policy against it: SFT teaches the format, a reward model turns a finite set of rankings into a reusable scalar function, and RL (PPO, or critic-free variants) reweights the policy’s own samples toward high reward. The reward model is a proxy for human judgement, so it is gameable, and the KL penalty to the frozen SFT policy is what bounds how far the policy can travel into the region where the proxy lies.

1. Why this exists

Start with what breaks. You have a pretrained base model and a nice instruction dataset, and you fine-tune on it with the ordinary causal-LM loss:

$${\mathcal{L}}_{\text{SFT}}(\theta )=-{\mathbb{E}}_{(x,y)\sim \mathcal{D}}\left[\sum _{t=1}^{|y|}\log {\pi }_{\theta }(y_t\mid x,y_{<t})\right]$$

• $x$ — the prompt. It is the conditioning context; without it you would be training an unconditional text model, which is why the dataset must pair every demonstration with the instruction that elicited it.
• $y$ — the demonstration: the completion a human wrote. It is the target of the maximum-likelihood fit, and therefore the ceiling on what SFT can teach.
• $y_t$ — the token at position $t$ of $y$.
• $y_{<t}$ — the tokens before position $t$; the prefix the model conditions on. In a causal transformer these are the same forward pass as $x$; the mask from Attention is what enforces the direction.
• $|y|$ — the token length of the demonstration. Note that the inner sum makes longer responses contribute more total loss, which is the first place a length bias enters the pipeline.
• ${\pi }_{\theta }(y_t\mid x,y_{<t})$ — the model’s probability for the single token that the human happened to write.
• $\mathcal{D}$ — the demonstration corpus.

Every symbol is doing ordinary language-modelling work, and that is the problem. Consider a concrete failure. You fine-tune a model on 20k demonstrations and it writes plausible, well-formatted answers. Then you hand it a request where the correct behaviour is to ask a clarifying question, because the request is ambiguous. There is no demonstration of that behaviour for this request, so the model does the nearest thing it has seen: it guesses and answers confidently. It does not do this because it is broken. It does this because the objective it was trained under has no term that distinguishes a good answer from a fluent one.

Four separable reasons, and it is worth being able to name them:

1. The loss is over token frequencies, not outcomes. A model that produces one excellent answer and one dangerous answer is penalised in proportion to how unlikely the dangerous sequence was in the demonstration data — not in proportion to the harm, and not at all if the dangerous sequence never appears.
2. There is no notion of preference between two adequate answers. Many completions are acceptable. MLE pushes mass at exactly one of them, the one in the dataset. This is the mechanical reason SFT models are often bland and over-hedged: the objective rewards converging on an average demonstration, not on being good.
3. Behaviours without demonstrations cannot be learned. Refusing a harmful request, saying “I don’t know”, asking for a clarification, declining to guess a number — none of these exist in a web-scale demonstration distribution as chosen behaviours. There is nothing to fit.
4. The model has no calibrated notion of its own uncertainty. Nothing in ${\mathcal{L}}_{\text{SFT}}$ distinguishes “confident and correct” from “confident and wrong”. Both are a sequence with a high likelihood.

The N+ RLHF reproduction paper states the mismatch in exactly these terms: the next-token-prediction objective “differs from the fundamental objective of ‘outputting contents that humans prefer’” (Huang et al. 2024, arXiv:2403.17031, §1).

The interview framing. “Why not just do more SFT?” is answered by point 3 and point 2 together. More demonstrations extend the distribution you are imitating; they do not create the ability to prefer. The moment you want the model to choose between two things it itself generated (which is where all the useful behaviour lives), you need a scalar and an optimiser, not more data.

2. How it works

2.1 The shape of the fix

Three ingredients, in order:

1. Define a scalar $r(x,y)\in \mathbb{R}$: how much a human prefers completion $y$ for prompt $x$.
2. Observe that $r$ is not available as a function. It is only available as comparisons — cheap, reliable, relative judgements (“A is better than B”) rather than absolute scores. The machinery has to turn comparisons into a differentiable reward.
3. Optimise the policy against that reward, anchored to the SFT model so the optimiser cannot wander into text the reward model has never seen and cannot judge.

Step 2 is the reason RLHF looks the way it does, and it is the thing candidates skip. Humans are far more consistent at “which of these two is better” than at “score this on a 1-7 scale”. The entire three-stage pipeline is the price of consuming only comparisons.

2.2 The three stages

flowchart TD
    P["Pretrained base LM<br/>(next-token prediction on web text)"]
    D1["Demonstrations<br/>(prompt, ideal answer) pairs"]
    S1["Stage 1: SFT<br/>supervised fine-tuning"]
    M["SFT model = reference policy π_ref"]
    D2["Rankings<br/>K samples per prompt, ordered by humans"]
    S2["Stage 2: Reward modelling<br/>K(K-1)/2 pairwise comparisons per prompt"]
    RM["Reward model r_φ<br/>scalar output"]
    P2["Prompts"]
    S3["Stage 3: RL<br/>PPO / GRPO against r_φ<br/>with per-token KL penalty to π_ref"]
    OUT["Aligned policy π_θ"]

    P --> S1
    D1 --> S1
    S1 --> M
    M --> S2
    D2 --> S2
    S2 --> RM
    M --> S3
    RM --> S3
    P2 --> S3
    S3 --> OUT
    OUT -. "re-sample with new policy,<br/>re-label, retrain RM" .-> D2

The dotted edge is the iterated / online variant, and it is not an optimisation detail — it is the main documented mitigation for reward over-optimisation (Gao, Schulman & Hilton 2022, arXiv:2210.10760). Anthropic’s HH work reports running exactly this: “an iterated online mode of training, where preference models and RL policies are updated on a weekly cadence with fresh human feedback data” (Bai et al. 2022, arXiv:2204.05862).

2.3 Stage 1 — SFT

You fine-tune the pretrained model with ${\mathcal{L}}_{\text{SFT}}$ on demonstrations of the behaviour you want. InstructGPT collected roughly 13k training prompts for this stage.

What it fixes: the format and register of the interaction. Answering rather than continuing; following an instruction rather than completing a document; producing the special tokens that mark a turn. Without SFT, a base model will happily continue your prompt as if it were a web page.

What it does not fix: everything in §1. In particular it cannot express preferences among model outputs, and it inherits every bias of the demonstration distribution — including the bias toward confident, verbose answers, because those are what labelers wrote.

Why it is still mandatory: the reward model is trained on the SFT model’s outputs, and the RL policy starts from the SFT model. SFT is what defines the distribution on which the reward model is even meaningful.

2.4 Stage 2 — Reward modelling

Sample several completions per prompt, have humans rank them, and fit a scalar function to the ranking. InstructGPT presents $K$ outputs per prompt, with $K$ ranging from 4 to 9, and trains on all $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs rather than only the adjacent ranks.

The model is the SFT model with the language-model head replaced by a scalar head, so the “reward” of a completion is one number read off the final token.

What it fixes: it converts a finite, expensive set of human judgements into a reusable scalar function that can be evaluated millions of times during RL. This is the economic point of the whole pipeline. Human labels are the scarce resource; a reward model is how you amortise them.

What it does not fix: it is a proxy. It is fit on a finite sample from a distribution that drifts as the policy changes, so maximising it eventually diverges from maximising true preference. That failure is the subject of Reward Modeling and it is the single most important thing to understand about stage 2.

2.5 Stage 3 — RL against the reward model

Use $r_{\varphi }$’s scalar output as the reward and optimise the policy, anchored to the SFT policy by a per-token KL penalty. The exact objective InstructGPT maximises (its Equation 2) is:

$$\text{objective}(\varphi )={\mathbb{E}}_{(x,y)\sim {\mathcal{D}}_{{\pi }_{\varphi }^{\text{RL}}}}[r_{\theta }(x,y)-\beta \log \frac{{\pi }_{\varphi }^{\text{RL}}(y\mid x)}{{\pi }^{\text{SFT}}(y\mid x)}]+\gamma {\mathbb{E}}_{x\sim {\mathcal{D}}_{\text{pretrain}}}[\log {\pi }_{\varphi }^{\text{RL}}(x)]$$

Every symbol, and what it is doing:

• ${\pi }_{\varphi }^{\text{RL}}$ — the policy being trained: the actor. $\varphi$ are its parameters. Note the paper uses $\varphi$ for the policy and $\theta$ for the reward model; the field has since mostly swapped those conventions, so always check which is which in a given paper.
• $r_{\theta }(x,y)$ — the reward model, parameters $\theta$. This is the only source of task signal in the objective. Remove it and there is nothing to optimise; scale it and you change the effective exchange rate against the KL term, which is why comparing $\beta$ across implementations is meaningless.
• $\beta$ — the KL reward coefficient: how much one nat of divergence from the reference costs the policy. It is a price, and like any price it should be read as a ratio to the reward scale. Remove it ($\beta =0$) and the policy walks out to whatever region maximises the proxy, which is the region where the proxy is least trustworthy; set it too high and the policy never moves.
• ${\pi }^{\text{SFT}}$ — the frozen SFT model, used as the reference. It appears in the denominator, so the log-ratio is zero at initialisation. That zero is what makes the penalty a distance rather than a penalty on being different from some arbitrary point. It must be frozen: if the reference moved with the policy, the KL term would be measuring nothing.
• $\gamma$ — the pretraining-loss coefficient, an InstructGPT-specific auxiliary term (the “PPO-ptx” variant). Warning: $\gamma$ normally means the discount factor in RL. InstructGPT overloads the symbol. $\gamma =0$ gives plain PPO.
• ${\mathcal{D}}_{\text{pretrain}}$ — the original pretraining corpus. The auxiliary term is a plain language-model loss mixed into the PPO gradients, and its purpose is to stop the model forgetting how to be a language model while it learns to be a helpful one.
• ${\mathcal{D}}_{{\pi }_{\varphi }^{\text{RL}}}$ — the distribution of prompts, with completions drawn from the current policy. This is what makes the stage online: the reward model scores what the policy actually produces, not what the SFT model produced.

Verified constants from InstructGPT Appendix C.4: $\beta =0.02$, $\gamma =27.8$, 256k RL episodes over roughly 31k unique prompts, iteration batch size 512 with minibatch 64.

The result the paper actually reports, and why it is the interesting one. InstructGPT found that increasing the KL coefficient to 2.0 — one hundred times the default — “the regressions still cannot be fixed”. The alignment tax (benchmark regressions on ordinary NLP tasks) was fixed by the pretraining mix term $\gamma$, not by a tighter leash on the KL. A bigger $\beta$ makes the policy move less; it does not make the movement less harmful per unit of distance. Hold onto that distinction; Reward Modeling returns to it.

2.6 The bandit framing, and why it matters downstream

InstructGPT is explicit that the RL environment is a bandit, not a multi-step MDP:

“The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode.” — Ouyang et al. 2022, §3.5

But the implementation generates token by token, and the reward arrives only at the end-of-sequence token. So the code runs a sequential sampler while the theory describes a one-step problem. That mismatch is the origin of most of the engineering in PPO for LLMs:

• The per-token KL penalty is applied at every token, so the immediate reward is not zero even though the RM reward is. The reward signal is dense in the KL and sparse in the objective.
• The value function has to predict the terminal return from every prefix, which makes it a “per-token reward model” rather than a planner.
• The discount factor $\gamma$ (in the RL sense, not InstructGPT’s) is set to $1.0$, because the episode is one bounded response and discounting would make the objective depend on where the answer happened to end.

Whether the bandit framing is the right framing is one of the questions the chapter on Monte Carlo, TD, and Search answers. The short version: it is exactly right for a single-turn, outcome-rewarded task, and it is wrong the moment there is a tool call, a multi-turn interaction, or a reward that depends on intermediate steps.

3. Implementation

3.1 The data schema is the design

Two datasets, two shapes, and getting them mixed up is a common source of silent garbage.

# Demonstrations (stage 1): one ideal answer per prompt.
{"prompt": "Explain what a KV cache is.", "response": "A KV cache stores ..."}

# Rankings (stage 2): K completions per prompt, ordered best to worst.
# The ordering is the label. Absolute scores are optional metadata.
{
    "prompt": "Explain what a KV cache is.",
    "responses": ["A KV cache stores ...",   # rank 0 (best)
                  "It caches keys ...",       # rank 1
                  "The KV cache is ...",      # rank 2
                  "K stands for keys ..."],   # rank 3 (worst)
    "ratings": [7, 6, 4, 2],                  # optional 1-7 quality, for margins
}

Two decisions are hidden in that schema and both matter:

• Rankings, not scores. The ordering carries the reliable signal. The ratings field is optional and is only worth collecting if you intend to use it (see the margin term in Reward Modeling).
• $K$ completions per prompt, not 1. A prompt with $K$ ranked responses yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairwise comparisons. At $K=4$ that is 6; at $K=9$ it is 36. The effective supervision per expensive annotation scales roughly quadratically in $K$ — it is the number of pairs, $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ — which is why labelers rank a group rather than judging one output at a time. InstructGPT notes that “the number of ranked pairs we trained the model on is an order of magnitude larger than the number of prompts” (§3.5).

The expansion from one ranking to pairs is where most people introduce a bug, so make it explicit and test the count:

from itertools import combinations


def expand_ranking(prompt: str, ranked_responses: list[str]) -> list[dict]:
    """Turn one K-way ranking into C(K,2) (prompt, chosen, rejected) triples.

    `ranked_responses` must be ordered best first. Every unordered pair is
    emitted once, with the earlier (better) entry as `chosen`. Emitting only
    adjacent ranks throws away most of the supervision; emitting ordered pairs
    both ways duplicates every example and halves the effective learning rate.
    """
    out = []
    for i, j in combinations(range(len(ranked_responses)), 2):
        out.append({
            "prompt": prompt,
            "chosen": ranked_responses[i],
            "rejected": ranked_responses[j],
        })
    return out


assert len(expand_ranking("p", ["a", "b", "c", "d"])) == 6   # C(4,2)
assert len(expand_ranking("p", ["a", "b"])) == 1             # C(2,2)

3.2 A runnable end-to-end toy: all three stages, in one file

The point of this snippet is not to be a production trainer; it is to make the three stages concrete and separated, and to demonstrate the one property of the reward model that causes the most confusion downstream: only reward differences are identified. It runs in a few seconds on CPU with torch>=2.0.

"""Three-stage RLHF in miniature: SFT -> Bradley-Terry RM -> policy gradient.

torch >= 2.0, CPU-only. Illustrative: the "LM" is a bag-of-tokens scorer with a
learned scalar head, not a transformer. The *structure* is the point.
"""
import itertools
import random

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
random.seed(0)

PHRASES = ["alpha", "beta", "gamma"]        # the "prompt" space
RESPONSES = ["good", "bad"]                 # the "completion" space
LENGTH = 8                                  # tokens per completion

# --- the "reward model": scores a completion; here, a linear scorer ----------
rm = nn.Linear(LENGTH, 1)


def encode(response: str) -> torch.Tensor:
    """Deterministic pseudo-token embedding so the demo needs no tokenizer."""
    h = torch.tensor([float(ord(c) % 7) for c in response]).repeat(LENGTH)[:LENGTH]
    return h / h.norm()


def reward(response: str) -> torch.Tensor:
    return rm(encode(response)).squeeze(-1)


# --- Stage 2: Bradley-Terry MLE on pairwise preferences ---------------------
# Ground truth the "humans" follow: "good" beats "bad".
preference_pairs = [
    (p, "good", "bad") for p in PHRASES for _ in range(64)
]

opt = torch.optim.Adam(rm.parameters(), lr=0.05)
for step in range(300):
    chosen = torch.stack([reward(c) for _, c, _ in preference_pairs])
    rejected = torch.stack([reward(r) for _, _, r in preference_pairs])
    # L = -E[ log sigmoid(r(chosen) - r(rejected)) ]   (see reward-modeling.md)
    loss = -F.logsigmoid(chosen - rejected).mean()
    opt.zero_grad()
    loss.backward()
    opt.step()

acc = (
    sum((reward(c) > reward(r)).item() for _, c, r in preference_pairs)
    / len(preference_pairs)
)
print(f"[stage 2] RM pairwise accuracy: {acc:.3f}")   # expect 1.0

# --- the invariance that matters: adding a constant to r changes nothing ----
with torch.no_grad():
    shift = torch.tensor(5.0)
    rm.bias.add_(shift)
    shifted_acc = (
        sum((reward(c) > reward(r)).item() for _, c, r in preference_pairs)
        / len(preference_pairs)
    )
# The RM loss and the RM's own accuracy are EXACTLY invariant to this shift.
# The RL objective is NOT: r enters the reward, and beta * KL is measured
# against zero. This is why the reward is bias-normalised before RL.
print(f"[stage 2] same accuracy after +{shift.item()} bias shift: {shifted_acc:.3f}")

# --- Stage 3: policy gradient against the (frozen) reward model -------------
# The policy is a logits table over two responses per prompt: a bandit.
policy_logits = nn.Parameter(torch.zeros(len(PHRASES), len(RESPONSES)))
policy_opt = torch.optim.Adam([policy_logits], lr=0.1)
init_probs = F.softmax(policy_logits.detach(), dim=-1)

BETA = 0.05      # KL coefficient: price of one nat of divergence from pi_ref
EPOCHS = 400

for epoch in range(EPOCHS):
    idx = torch.arange(len(PHRASES))
    probs = F.softmax(policy_logits, dim=-1)                  # pi_theta
    dist = torch.distributions.Categorical(probs=probs)
    action = dist.sample()                                     # a_t ~ pi_theta
    r = torch.stack([
        reward(RESPONSES[a]) for a in action.tolist()
    ]).detach()
    logp = dist.log_prob(action)                               # log pi_theta(a|s)
    logp_ref = torch.log(init_probs[idx, action]).detach()      # log pi_ref(a|s)
    kl_t = (logp - logp_ref).detach()                          # single-sample KL
    total_r = r - BETA * kl_t                                  # reward + KL penalty
    loss = -(logp * total_r).mean()                            # REINFORCE
    policy_opt.zero_grad()
    loss.backward()
    policy_opt.step()

with torch.no_grad():
    final = F.softmax(policy_logits, dim=-1)
print(f"[stage 3] P(response='good') before: {init_probs[0, 0]:.3f} "
      f"after: {final[0, 0]:.3f}")

Three facts this toy makes visible, all of which generalise:

1. The RM only ever sees differences. After the bias shift, pairwise accuracy is unchanged and the RM loss is unchanged, because $\sigma (r_w-r_l)=\sigma ((r_w+c)-(r_l+c))$. That is Bradley-Terry’s shift invariance, strictly.
2. The RL objective is not invariant to that shift, because the reward enters the objective directly while the KL term is measured against zero. Hence the normalisation step in §3.3.
3. The KL term is what keeps the final policy away from a corner. Set BETA = 0.0 in the snippet and the policy drives $P(\text{good})\to 1$ regardless of how sharp the RM is; set BETA = 5.0 and it barely moves from $0.5$. There is nothing subtle about it once you can turn the dial yourself.

3.3 The implementation details that change results

(a) Reward bias normalisation, before RL. Because of the invariance in point 1 above, the reward’s origin is arbitrary. InstructGPT: “since the RM loss is invariant to shifts in reward, we normalize the reward model using a bias so that the labeler demonstrations achieve a mean score of 0 before doing RL.” The N+ reproduction implements it literally — iterate the SFT dataset, score the reference responses, and set the mean as the reward head’s bias. The shift in the toy above is exactly the thing this removes.

(b) Reward whitening, during RL. Two subtly different operations, and the asymmetry between them is deliberate: rewards are standardised without restoring the mean (shift_mean=False), while advantages are standardised with the mean restored. Restoring the mean on rewards would add a constant to every return, and a constant added to every return changes nothing in the policy gradient; restoring it on advantages preserves the sign structure while rescaling the magnitude. See the N+ reproduction, Details 24 and 25.

(c) Rollout temperature. The N+ reproduction samples at temperature 0.7. Sampling temperature is not a preference-alignment knob; it is a coverage knob. Too low and the policy never emits the samples that the reward model would penalise, so RL cannot correct anything; too high and most of the batch is low-quality text that teaches the RM nothing useful.

(d) KL estimator. The single-sample estimate $\log {\pi }_{\theta }(a_t\mid s_t)-\log {\pi }_{\text{ref}}(a_t\mid s_t)$ is unbiased for the per-token KL but high variance. The k3 estimator (${\pi }_{\text{ref}}/{\pi }_{\theta }-\log ({\pi }_{\text{ref}}/{\pi }_{\theta })-1$) is non-negative and lower-variance, and is what most modern frameworks default to (Schulman 2020; veRL exposes kl_penalty: kl | abs | mse | low_var_kl | full).

(e) The order of stages is not optional for the RM. The reward model is trained on completions from the SFT policy. Train it on a base model’s outputs and then RL against it from an SFT-initialised policy and you have trained a reward model on the wrong distribution — the single most common way to build a pipeline that “works” but produces garbage.

4. Numbers and trade-offs

4.1 What the stages cost

The three stages have completely different bottlenecks, which is why they are usually run as separate jobs with separate hardware.

The RL number is the one people underestimate. OpenRLHF’s README states the design premise plainly: “RLHF training spends 80% of the time on sample generation.” That is a project self-report, not a controlled measurement, but the qualitative claim is forced by the algorithm: PPO is on-policy, so every gradient step needs fresh samples from the current policy, and generating a 256-to-32768-token response costs more than scoring it. Everything in PPO for LLMs and GRPO that removes a model does not remove the rollout; only the asynchronous systems in RL Frameworks attack that term directly.

4.2 The four models in memory

RLHF’s stage 3 needs four things resident, and this table is the reason the rest of the part is organised around cost reduction.

The dominance of this cost is why all three of the next chapters are about removing one of these rows:

flowchart LR
    A["PPO<br/>actor + critic + RM + reference"]
    B["GRPO<br/>actor + RM + reference<br/>(critic removed: group mean baseline)"]
    C["DPO<br/>actor + reference<br/>(RM and critic removed: closed-form reparameterisation)"]
    A -->|"baseline = mean reward<br/>over G samples"| B
    B -->|"preference objective solved<br/>in closed form"| C

The honest reading of that diagram is not “C is best”. It is: each arrow trades a resource for a different one. GRPO pays $G\times$ the rollout tokens for the baseline it removed. DPO pays for the reward model by requiring that the preference data already exist, offline, and it cannot explore. See GRPO and DPO respectively.

4.3 The alignment tax, and where it actually goes

RLHF improves preference ratings and degrades some ordinary benchmarks (SQuAD, DROP, translation, summarisation) — the “alignment tax”. InstructGPT’s finding is that the fix is not a tighter KL budget:

• Raising $\beta$ from 0.02 to 2.0 (100x) did not recover the regressions.
• Adding the pretraining-log-likelihood term with $\gamma =27.8$ did.

Read that as: the regressions come from the policy leaving the language-modelling objective entirely, not from it travelling too far. A KL penalty limits distance; it does not supply the missing objective. If you only remember one number from this section, remember that you cannot buy back a lost capability with a bigger $\beta$.

4.4 Named hyperparameter ranges

Taken from the N+ RLHF reproduction (TL;DR summarisation) so that the ranges in this book are attested rather than invented:

Compare InstructGPT: $\beta =0.02$, batch 512, minibatch 64, 256k episodes. The tenfold difference in $\beta$ between two implementations of the same algorithm is normal: $\beta$’s meaningful value depends on the reward scale and the normalisation scheme, so it is not transferable. Do not quote a $\beta$ without quoting the scheme.

5. Pitfalls

Training the reward model on the wrong distribution. You collect preference data by sampling from a base model (or a different model), train the RM, then run RL from an SFT-initialised policy. Symptom: the RM’s held-out accuracy looks fine, but during RL the reward rises immediately and the outputs get worse. The policy has walked off the RM’s training distribution in the first hundred steps.

Forgetting reward bias normalisation. Symptom: the KL term consumes the entire reward at the start of RL, so the policy is pulled back toward the reference while the RM signal is too small to move it; the reward curve is flat and the KL curve is huge. The origin of the reward is arbitrary (Bradley-Terry identifies only differences), so it must be centred against the SFT responses before RL.

Reading $\gamma$ as the discount factor in the InstructGPT objective. It is the pretraining-mix coefficient there, and it is 27.8 — a number that makes no sense as a discount. Symptom of confusion: you “tune the discount” and change nothing, or you set it to 0.99 and silently drop the auxiliary LM loss.

Believing more SFT is a substitute for RL. Symptom: you scale demonstrations 10x, the model gets better at style and no better at refusing, clarifying, or hedging appropriately; and eval scores on honesty-flavoured benchmarks plateau. Demonstrations cannot express a preference between two model outputs that do not exist in the dataset.

Treating the RM as ground truth. Symptom: proxy reward climbs smoothly while a held-out human or gold-model score flattens and then falls. This is the signature of over-optimisation, and it is the subject of the next chapter.

Using hand-written reward functions (length, ROUGE, sentiment) as a shortcut. Symptom: the policy discovers the exact thing you scored. Rewarding length produces long answers; rewarding ROUGE produces extractive copying; rewarding sentiment produces sycophancy. A hand-written reward is a reward model with no training data, and therefore with all of the hackability and none of the calibration.

Scoring with the RM at a different context length than it was trained on. Symptom: rewards drift with position and long responses score erratically. The reward is read off the final token; if the RM was trained on sequences of a certain length distribution, truncating or padding at rollout time changes the statistics of the very number you are optimising.

Forgetting that the preference data has a producer. Symptom: the RM systematically prefers whatever the labeler pool systematically prefers — longer, more confident, more list-shaped, more apologetic. Nothing in the loss corrects for this; the only fixes are labeler instructions and a diverse annotation pool.

6. Using it in a project

6.1 The decision order

Work down this list. Stop at the first line that applies.

1. Is the desired behaviour checkable? Math answer match, unit tests, a schema validator, a formal verifier. Then use RLVR — reinforcement learning with verifiable rewards — instead of a learned reward model. It is cheap, deterministic, and unfakeable with respect to the property it checks. Tulu 3 introduces the term and the recipe (Lambert et al. 2024, arXiv:2411.15124). This is the dominant pattern for reasoning post-training, and it is what GRPO is built to consume. Note that “unfakeable” is scoped: it moves the hackable surface into the verifier’s blind spots (regex-format exploits, test-case overfitting).
2. Do you already have a good pile of preference pairs, and no environment? Then start with DPO and skip the RL loop entirely (DPO). It is a classification loss on pairs; no sampling, no RM, no critic. It is the right default for style, tone, safety and general preference alignment.
3. Do you have prompts, a way to score a completion, but no pairs? Then you need a reward model, and you are in this part: stages 2 and 3. Prefer a critic-free RL method (GRPO/RLOO) unless the reward is genuinely dense.
4. Is the task multi-turn with tools? Then the bandit framing is wrong, the credit-assignment problem is real, and you are in the agentic-RL setting discussed at the end of Monte Carlo, TD, and Search.

6.2 When NOT to run RLHF

• When preference data is too thin. The RM is fit on your comparisons. With a few hundred pairs you will fit an RM that is worse than the SFT model at everything except ranking that specific dataset, and RL will optimise the difference between them. Below roughly a few thousand comparisons, prefer a hand-written checker (RLVR) or do nothing: a well-instruction-tuned SFT model is usually better than a badly-RLHF’d one.
• When the task is verifiable. Do not pay for a reward model, a critic, and an annotation pipeline to learn something a string comparison already knows.
• When you cannot afford rollouts. RL needs fresh generations from the current policy, repeatedly. If your inference budget is one pass per prompt, use offline preference optimisation.
• When the behaviour is a policy question, not a preference question. “Never output this API key format” is a filter. “Always cite sources” is a prompt. Alignment machinery is for things you can only express as “this output is better than that one”.
• When you cannot evaluate. If you have no held-out, frozen preference signal, you cannot tell over-optimisation from progress, and you will ship the run that looks best on the proxy. That is worse than not running RL.

6.3 The cheap version people forget

RLAIF / Constitutional AI replaces the human preference labels with labels produced by a model following explicit written criteria (Bai et al. 2022, arXiv:2212.08073; Lee et al. 2023, arXiv:2309.00267). This removes the annotation cost, not the reward-modelling cost, and it does not make the reward model a better proxy — it makes it a proxy for a written spec instead of for a labeler pool. That is often a genuine improvement (the spec is inspectable and editable), and it is also a new attack surface (the model’s reading of the spec is now the ceiling).

7. How you would prove it worked

7.1 The measurement that is actually hard

Preference alignment has no single metric, and every candidate metric is gameable. The defensible structure is a frozen, held-out preference signal that is never used for training or model selection, plus a report of the gap between that signal and the training reward.

7.2 The ablation that answers “did RL do anything”

Run three arms from the identical SFT checkpoint and identical prompt set:

1. SFT only — the baseline, and the arm most reports omit.
2. SFT + offline preference optimisation (DPO/SimPO) on the same pairs.
3. SFT + RL against the RM trained on those same pairs.

(example) A common outcome in small-scale runs is that arms 2 and 3 land within noise of each other on held-out win-rate, while arm 3 costs an order of magnitude more compute and is far less stable. The result is not a failure; it is the finding. Report the compute alongside it, and report the length of the responses in each arm, because a win-rate improvement that comes entirely with a simultaneous 40% length increase is usually a length effect in disguise.

Reporting template (example — replace the numbers). “We trained an RM on 12k comparison pairs, then ran PPO for 1.5k steps at $\beta =0.05$. Held-out human win-rate against SFT rose from 50% to 61% (example) over the first 600 steps, then flattened while proxy reward continued to climb; mean response length rose 22% (example) over the same window. Gold win-rate began falling at step 900 (example), which we read as the over-optimisation onset predicted by the reward-KL scaling law; we shipped the step-600 checkpoint.”

8. Interview drill

“What does each stage of RLHF contribute, and what happens if you remove it?”

Strong answer: SFT fixes the format and register and defines the distribution the rest of the pipeline lives on. The reward model converts a finite set of expensive comparisons into a reusable scalar that can be evaluated during RL. RL reweights the policy’s own output distribution toward high reward, which is the only stage that can discover behaviour no human demonstrated. Remove SFT and the reward model is trained on a distribution you do not want; remove the RM and there is nothing to optimise; remove RL and you are back to imitation, with no ability to prefer.

Common wrong answer: “SFT teaches the model, then RL makes it safer.” Safety is one axis; the structural point is imitation vs preference optimisation and off-policy data vs on-policy samples.

“Where does the preference data come from, exactly?”

Strong answer: from the model’s own samples. You sample $K$ completions per prompt (InstructGPT: $K$ from 4 to 9), have humans rank them, and train on all $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs. Human reliability is much higher on relative judgements than on absolute scores, which is why rankings rather than Likert scores drive the loss. In the iterated variant you re-sample with the current policy and re-label on a cadence, which is the documented mitigation for over-optimisation.

Follow-up: “why all pairs, not just best vs worst?” Because a $K$-way ranking contains $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ comparisons for the price of one annotation, and because during RL the policy’s samples concentrate near its mode — which is the middle of the quality distribution, not the extremes. Training only on best-vs-worst teaches the RM about a region the policy rarely visits.

“Why is the KL penalty mandatory?”

Strong answer: three reasons. It bounds how far the policy can travel from the distribution the RM is valid on; it preserves language quality, because off the SFT distribution the RM’s judgement is arbitrary and unconstrained maximisation produces degenerate text; and it keeps a floor under the policy’s support, which prevents the mode collapse that sequential generation would otherwise drive toward. Trap: claiming the KL penalty prevents reward hacking. Gao et al. report that it raises the proxy score achievable at a given KL distance but does not improve the gold score-KL frontier: it is a distance limiter, not a cure.

“What is the alignment tax, and how do you fix it?”

Strong answer: the drop on ordinary capability benchmarks after alignment. InstructGPT’s result is that it is fixed by mixing the pretraining loss back in ($\gamma =27.8$), not by tightening the KL budget — raising $\beta$ 100x did not recover the regressions. The interpretation is that the policy has left the language-modelling objective, and a distance constraint cannot restore an objective.

Common wrong answer: “increase the KL penalty until the regressions go away.” Why wrong: a bigger $\beta$ makes the policy move less far, not less harmfully per unit distance. It buys you a smaller alignment tax by buying you a smaller alignment gain.

“InstructGPT calls this a bandit. So why does your PPO code have a value function and a discount factor?”

Strong answer: the objective is a one-step problem — prompt in, response out, reward at the end — so there is no future state to plan over and the critic is purely a variance-reduction device. But the sampler is sequential, one token at a time, so implementations keep GAE over the token sequence with $\gamma =1$ and $\lambda =0.95$ as a within-response bootstrap. The value head is effectively predicting the terminal RM score from each prefix. That is also why the critic is hard to fit and why critic-free methods are attractive.

Follow-up: “when is the bandit framing actually wrong?” The moment there is a tool call, a multi-turn interaction, or a reward that depends on intermediate steps. Then you have a genuine multi-step decision problem and credit assignment across turns becomes the central difficulty.

“Would you run RLHF for a coding assistant?”

Strong answer: probably not as the first move. Code is largely verifiable, so RLVR with unit tests is cheaper and more trustworthy than a learned RM. Preference modelling still earns its place for the style axis — readable diffs, good commit messages, explaining a change — which is not checkable. The realistic pipeline is both: verifiable rewards for correctness, preference optimisation for taste.

Next: Reward Modeling — the Bradley-Terry model turns those rankings into a differentiable scalar, and that scalar is the thing the policy will learn to game.