DPO

TL;DR — Direct Preference Optimization removes the reward model and the RL loop from RLHF, and turns preference learning into a binary classification loss on pairs. The derivation is three algebraic moves: solve the KL-regularised reward-maximisation problem in closed form, invert that solution to express the reward as $r(x,y)=\beta \log \frac{{\pi }_{\theta }(y|x)}{{\pi }_{ref}(y|x)}+\beta \log Z(x)$, then substitute it into Bradley–Terry — where the prompt-only term $\beta \log Z(x)$ cancels because preferences use reward differences. What is left is $-\log \sigma$ of the gap between the winner’s and loser’s implicit rewards, i.e. one forward pass over each response, no sampling, no critic, no reward model. $\beta$ plays a dual role: KL penalty coefficient and temperature. DPO still needs a frozen reference copy, and it is not the tool for verifiable tasks — that is RLVR’s job.

1. Why this exists

By the end of the PPO chapter, RLHF is an assembly line: a reward model to turn comparisons into a scalar, a value model to reduce the variance of the policy gradient, a reference model for the KL leash, and a sampling loop that generates fresh completions on every step. Four models, a rollout engine, and a set of stability problems — clipping, advantage estimation, train/inference log-probability mismatch — that only exist because of the RL loop.

The GRPO chapter removes one of those four models. DPO asks a different question: if the reward model is just a reparameterisation of the optimal policy, why train it at all? Rafailov et al. 2023, Direct Preference Optimization: Your Language Model is Secretly a Reward Model (arXiv:2305.18290), answers that for the KL-regularised objective, the optimal policy and the reward function are related by a one-to-one transformation. The policy can therefore be the reward model, and the RLHF objective can be optimised directly on preference pairs with supervised learning.

The concrete failure DPO fixes is economic rather than mathematical. Suppose you have 50,000 preference pairs and no environment. PPO and GRPO both require sampling from a policy in a loop; if sampling or verifiable scoring is unavailable or too expensive, that data sits unused. DPO uses it as a static classification dataset:

• No reward model. The policy parameterises the reward implicitly.
• No sampling loop. Training is a forward and backward pass over pairs.
• No critic, no GAE, no advantage estimation, no clip ratio. The loss is $\log \sigma$ over a difference of log-probabilities.
• No RL stability problems. A supervised loss converges; you can use a normal optimizer, a normal schedule, and early stopping that means what it says.

What you give up is exploration: DPO can only reweight the behaviours present in its preference data, because it never generates anything. That single sentence explains most of the comparison with online RL in Section 6.

The one-line framing. DPO is “your language model is secretly a reward model” turned into a loss: train the policy so that its log-ratio to the reference orders the winner above the loser.

2. How it works

2.1 Step 0 — the objective both methods are solving

Every RLHF method starts from the same constrained problem. For a prompt $x$ drawn from a dataset, maximise reward while staying near a frozen reference policy:

$$\underset{{\pi }_{\theta }}{\max }{\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[r(x,y)]-\beta D_{KL}[{\pi }_{\theta }(y\mid x)\Vert {\pi }_{\text{ref}}(y\mid x)]$$

• $x$ — a prompt; $y$ — a completion; $\mathcal{D}$ — the prompt distribution.
• $r(x,y)$ — the (unknown) true reward. DPO never instantiates this function; that is the whole point.
• ${\pi }_{\text{ref}}$ — the frozen reference policy, usually the SFT checkpoint. It anchors the optimisation and defines what “near” means.
• $\beta$ — the KL coefficient: how many units of reward one nat of divergence costs. In DPO it is the same number that also appears as the temperature of the optimal policy (Section 2.6).
• $D_{KL}$ — the divergence penalty. Without it the maximum is a point mass on the single highest-reward completion; with it, the optimum stays a full distribution whose shape depends on $\beta$.

Remove the KL and the optimal policy is degenerate — exactly the mode collapse the PPO chapter describes. Keep it, and there is a closed-form solution, which is the crack DPO drives through.

2.2 Step 1 — solve it in closed form

Fix $x$ and drop it from the notation. Define the partition function

$$Z(x)=\sum _y{\pi }_{\text{ref}}(y\mid x)\exp (\frac{1}{\beta }r(x,y))$$

and the tilted policy

$${\pi }_r(y\mid x)=\frac{1}{Z(x)}{\pi }_{\text{ref}}(y\mid x)\exp (\frac{1}{\beta }r(x,y))$$

• $\exp (r(x,y)/\beta )$ — an exponential tilt of the reference: completions with higher reward get geometrically more mass. $\beta$ in the denominator is what makes it a temperature: small $\beta$ concentrates hard on the argmax, large $\beta$ leaves the reference nearly untouched.
• $Z(x)$ — the partition function, the normaliser that makes ${\pi }_r$ a distribution. It is a sum over every possible completion, so it is intractable to compute — and DPO is designed so it never has to be.
• ${\pi }_r$ — the optimal policy for the KL-regularised problem.

Now rearrange the definition of ${\pi }_r$ into the form DPO needs later. Take logs:

$$\log \frac{{\pi }_r(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}=\frac{1}{\beta }r(x,y)-\log Z(x)$$

which solves for the reward:

$$r(x,y)=\beta \log \frac{{\pi }_r(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}+\beta \log Z(x)$$

This is the pivotal equation. It says the reward is recoverable from the policy plus a prompt-dependent constant. Substitute it back into the objective to see why ${\pi }_r$ is the maximiser: the objective becomes $\beta \log Z(x)-\beta D_{KL}({\pi }_{\theta }\Vert {\pi }_r)$, so it is maximised exactly at ${\pi }_{\theta }={\pi }_r$, and the KL term is what makes the closed form exist at all.

2.3 Step 2 — the implicit reward

DPO’s move is to treat the parameterised policy as if it were optimal and read the reward off it. Define

$${\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$$

• ${\hat{r}}_{\theta }(x,y)$ — the implicit reward of the trained policy. It is the log-probability ratio to the reference, scaled by $\beta$. Because the same quantity appears in the closed-form solution, training ${\pi }_{\theta }$ to fit preferences is training this reward—there is no separate head.
• ${\pi }_{\theta }(y\mid x)$ — the policy’s likelihood of the completion under training. Differentiable, so gradients flow into it.
• ${\pi }_{\text{ref}}(y\mid x)$ — the frozen reference likelihood, a constant with respect to $\theta$. This is why DPO needs a second forward pass (or a cached lookup) per sequence: the reward is relative, not absolute.
• $\beta \log Z(x)$ — dropped, deliberately. It depends only on the prompt. Bradley–Terry scores a pair $(y_w,y_l)$ for the same $x$, and only reward differences are identified, so a prompt-constant term cancels exactly. This is the step that makes DPO tractable: the intractable normaliser never has to be computed.

Why does the same policy that generates text also act as a reward model? Because the optimal solution ties them: a completion is “good” exactly to the extent that the optimal policy would produce it more often than the reference would. DPO takes that statement as the definition and fits it.

2.4 Step 3 — substitute into Bradley–Terry

Preferences are modelled by Bradley–Terry: for a winner $y_w$ and loser $y_l$ under the same prompt $x$,

$$p(y_w\succ y_l\mid x)=\sigma (r(x,y_w)-r(x,y_l)),\sigma (z)=\frac{1}{1+e^{-z}}$$

• $y_w,y_l$ — the preferred and dispreferred completions of the pair.
• $r(x,y_w)-r(x,y_l)$ — only the difference matters, which is exactly the property that lets $\beta \log Z(x)$ vanish.
• $\sigma$ — the logistic function, turning a real-valued score gap into a probability in $(0,1)$.

Substituting the implicit reward from Step 2 into the difference:

$$p(y_w\succ y_l\mid x)=\sigma (\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)})$$

The $Z(x)$ terms cancel. What remains contains only quantities you can compute: log-probabilities under two models.

2.5 Step 4 — the loss

Take the negative log-likelihood over the dataset. This is DPO:

$$L_{\text{DPO}}({\pi }_{\theta };{\pi }_{\text{ref}})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \right(\beta \log \frac{{\pi }_{\theta }(y_w\mid x)}{{\pi }_{\text{ref}}(y_w\mid x)}-\beta \log \frac{{\pi }_{\theta }(y_l\mid x)}{{\pi }_{\text{ref}}(y_l\mid x)}\left)\right]$$

Read it as a classification problem: the “logit” is the gap between the winner’s and loser’s implicit rewards, the label is always “winner first”, and the loss is binary cross-entropy. Maximising it pushes the implicit reward gap positive.

2.6 Step 5 — the gradient, and what it means

Differentiating (the paper’s Equation 8):

$${\nabla }_{\theta }{L}_{\text{DPO}}=-\beta {\mathbb{E}}_{(x,y_w,y_l)}\left[\underset{\text{how wrong the current ranking is}}{\underbrace{\sigma ({\hat{r}}_{\theta }(x,y_l)-{\hat{r}}_{\theta }(x,y_w))}}\right({\nabla }_{\theta }\log {\pi }_{\theta }(y_w\mid x)-{\nabla }_{\theta }\log {\pi }_{\theta }(y_l\mid x)\left)\right]$$

• The bracket is a contrastive update: increase the log-likelihood of $y_w$, decrease that of $y_l$. If that were the whole story, DPO would be a fancy ranking loss; the weight is what makes it adaptive.
• $\sigma ({\hat{r}}_{\theta }(y_l)-{\hat{r}}_{\theta }(y_w))$ — the paper’s own annotation is “higher weight when reward estimate is wrong”. If the model already ranks the winner far above the loser, the sigmoid is near $0$ and the pair contributes almost no gradient. If it has them backwards, the sigmoid is near $1$ and the pair gets a full-strength correction. DPO automatically spends its gradient budget on the pairs it is currently getting wrong.
• $\beta$ multiplies the whole update, so it scales the effective learning rate on the implicit reward — one more reason its value is not cosmetic.

There is no sampling and no RL loop: every quantity in the gradient is a log-probability of data you already have.

2.7 The dual role of $\beta$

$\beta$ appears in two places and both are the same number:

• In the RLHF objective, $\beta$ is the KL penalty coefficient — the price of diverging from ${\pi }_{\text{ref}}$.
• In ${\pi }_r\propto {\pi }_{\text{ref}}\exp (r/\beta )$, $\beta$ is a temperature: small $\beta$ concentrates the optimal policy on the highest-reward completion; large $\beta$ keeps it near the reference.

Equivalently, in $L_{\text{DPO}}$ the argument of the sigmoid is $\beta \cdot (\Delta \log {\pi }_{\theta }-\Delta \log {\pi }_{\text{ref}})$, so $\beta$ sets the sharpness of the sigmoid and the units of the implicit reward. The practical consequences are in Section 5; the structural consequence is that $\beta$ is not transferable between DPO and its length-normalised relatives (SimPO rescales the implicit reward, so the same $\beta$ means a different thing).

Published values are typically in the $0.01$–$0.5$ range with $0.1$ a common starting point. [UNVERIFIED: the DPO paper’s per-task default $\beta$ — the role and position of $\beta$ are verified from the paper, but its hyperparameter appendix was not extracted. Treat 0.1 as a starting point, not as “the paper’s value”.]

2.8 The picture

flowchart LR
    X["prompt x"] --> W["winner y_w"]
    X --> L["loser y_l"]
    W --> LW["log π_θ(y_w|x) − log π_ref(y_w|x)"]
    L --> LL["log π_θ(y_l|x) − log π_ref(y_l|x)"]
    LW --> D["implicit-reward gap × β"]
    LL --> D
    D --> S["σ(·)"]
    S --> NLL["−log σ → gradient step"]
    NLL --> X

No reward model, no rollout, no value function. One differentiable forward pass per sequence and one frozen reference forward pass (usually cached once).

3. Implementation

3.1 TRL DPOTrainer

TRL ships DPO [as of 2026-10], TRL v1.15.0. The dataset is pairs of (prompt, chosen, rejected):

# trl >= 1.15.0, transformers, torch
from datasets import load_dataset
from trl import DPOConfig, DPOTrainer

dataset = load_dataset("argilla/ultrafeedback-binarized-preferences-cleaned", split="train")

config = DPOConfig(
    output_dir="dpo-ultrafeedback",
    beta=0.1,                       # KL coefficient == implicit-reward temperature
    loss_type="sigmoid",            # the loss in Section 2.5
    max_length=2048,                # prompt + completion, both sequences
    max_prompt_length=512,
    precompute_ref_log_probs=True,  # cache π_ref logprobs once; frees the reference
    learning_rate=5e-7,             # DPO likes a much smaller lr than SFT
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    num_train_epochs=1,             # 1 epoch is usually enough; more overfits
    bf16=True,
)

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    ref_model=None,                 # None + PEFT means the adapter is disabled for π_ref
    args=config,
    train_dataset=dataset,
)
trainer.train()

What the knobs do:

• beta — the KL coefficient and reward temperature. Lower $\beta$ lets the policy move further from the reference and sharpens the sigmoid; higher $\beta$ keeps it closer. This is the first knob to ablate.
• loss_type — which offline preference loss to run. sigmoid is the DPO loss derived above; the same trainer hosts the variants in Section 3.2.
• precompute_ref_log_probs — compute ${\pi }_{\text{ref}}$’s log-probabilities once and store them, so training does not need a live reference model. This is the single most effective memory optimisation for DPO: the reference is frozen, so its contribution to the loss is a constant.
• learning_rate — DPO trains with a far smaller lr than SFT; the update in Section 2.6 is already scaled by $\beta$ and by a sigmoid weight.
• num_train_epochs — one. With an offline pair dataset, extra epochs fit the pairs rather than the preference (Section 5).

Field names move between releases; check them against the installed version if a run fails at construction.

3.2 The offline preference family

All of these are one loss_type (or one trainer) away in the same libraries. Each exists because DPO has a specific weakness:

Also worth a line: TR-DPO (2404.09656) updates the reference during training; Nash Learning from Human Feedback (2312.00886) targets a Nash equilibrium instead of Bradley–Terry MLE; Self-Rewarding Language Models (2401.10020) and SPIN (2401.01335) generate their own preference data. $\beta$ is not the same $\beta$ across these — normalisation differs, so cross-paper hyperparameter comparisons are meaningless.

3.3 The loss from memory

import torch

def dpo_logits(logp_w, logp_ref_w, logp_l, logp_ref_l, beta):
    """Each argument is a (B,) sum of log-probabilities over the sequence.
    Returns the (B,) 'logit' whose sigmoid is P(y_w ≻ y_l | x)."""
    implicit_w = beta * (logp_w - logp_ref_w)     # r̂_θ(x, y_w)
    implicit_l = beta * (logp_l - logp_ref_l)     # r̂_θ(x, y_l)
    return implicit_w - implicit_l                # Z(x) cancels: same prompt

def dpo_loss(logp_w, logp_ref_w, logp_l, logp_ref_l, beta, label_smoothing=0.0):
    logits = dpo_logits(logp_w, logp_ref_w, logp_l, logp_ref_l, beta)
    # -log σ(z) computed stably: softplus is log(1 + e^z)
    loss = torch.nn.functional.softplus(-logits)
    if label_smoothing:
        loss = (1 - label_smoothing) * loss + label_smoothing * torch.nn.functional.softplus(logits)
    return loss.mean()

Two notes. Use softplus rather than -torch.log(torch.sigmoid(z)); the composed form underflows for large negative $z$, which happens routinely once $\beta$ is large or the implicit-reward gap is wide. And the log-probabilities must be summed over the completion only, not over the prompt — including prompt tokens adds a common constant to both sides and silently weakens the signal.

4. Numbers and trade-offs

4.1 What DPO does and does not cost

The headline is the row that is empty: DPO generates nothing. That is its advantage in cost and stability and its limitation in capability, and the same fact explains both. [as of 2026-10] The reference’s log-probabilities can be precomputed once, which removes the reference model from the live working set — so the memory picture for DPO is “one trainable model plus a disk cache”.

4.2 $\beta$ in practice

• $\beta$ too small: the sigmoid argument is scaled down, so the implicit reward gap needed to saturate the loss grows; the loss saturates (gradients vanish) and the policy is free to drift far from the reference, which is where degenerate outputs live.
• $\beta$ too large: the loss is flat in the direction that matters because the model can barely move relative to the reference; training “succeeds” with almost no behavioural change.

(example) A reasonable sweep is $\{0.01,0.05,0.1,0.3\}$ on your held-out pairwise accuracy, and the useful diagnostic is not the training loss but the mean implicit-reward gap and the mean KL from the reference over training. If KL is near zero after a full epoch, $\beta$ is too large for your data; if KL is enormous and generations degrade, it is too small.

4.3 Data is the binding constraint

DPO cannot repair a weak preference signal, and it cannot explore to find one. If your pairs are noisy, small, or unrepresentative of the policy you are aligning, DPO fits the noise. The comparison paper Xu et al. 2024, Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (arXiv:2404.10719, ICML 2024), and Ahmadian et al. 2024, Back to Basics (arXiv:2402.14740), disagree about the ranking but agree on this: both methods are dominated by the quality and coverage of the comparison data.

5. Pitfalls

$\beta$ too small. Symptom: the loss drops to near zero within a few hundred steps and then the implicit-reward gap explodes; generations become repetitive or degenerate, and the KL from the reference grows without bound. Cause: the sigmoid is saturated, so the gradient is vanishing while the policy is unconstrained. Fix: raise $\beta$, shorten training, and monitor KL.

$\beta$ too large. Symptom: the loss barely moves and the aligned model is indistinguishable from SFT in blind comparisons; the mean implicit-reward gap stays near zero. Cause: the update is too small relative to the reference anchor. Fix: lower $\beta$ and/or raise the learning rate, and evaluate against the SFT baseline rather than the loss.

DPO reduces the likelihood of both responses. Symptom: the win-rate improves while the model’s absolute log-likelihood on both chosen and rejected completions falls; perplexity on the preference set rises. Cause: the loss depends only on the difference of implicit rewards, so the cheapest way to satisfy it can be to push ${\pi }_{\theta }(y_l)$ down harder than ${\pi }_{\theta }(y_w)$ up — the gradient in Section 2.6 has a minus sign in front of the loser term and nothing pins the winner’s absolute probability. Fix: report both likelihoods, not just the gap; use a length-normalised variant (SimPO), a monolithic loss (ORPO), or add an SFT/regularisation term. [UNVERIFIED: which specific implementations add such a term by default — check the trainer’s config before claiming DPO “usually” does this.]

Length inflation. Symptom: mean response length climbs; the model becomes verbose. Cause: the implicit reward is a sum of per-token log-probabilities, so longer sequences have larger-magnitude rewards and the objective is not length-neutral; this is SimPO’s stated motivation for using the average log-prob instead. Fix: SimPO, or an explicit length term.

Overfitting the pairs. Symptom: held-out pairwise accuracy on the training distribution keeps rising while a fresh judge or human evaluation stops improving, or falls; the model’s outputs converge on the idiosyncratic style of the chosen set. Cause: one offline dataset, many epochs. Fix: one epoch, early stopping on an evaluation from a different source than the training pairs, and IPO’s fixed-margin behaviour if the preference is genuinely deterministic.

A reference that is the same as the policy at step 0. Symptom: the loss starts at $\log 2$ and does not move, no matter the learning rate. Cause: ${\hat{r}}_{\theta }=\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})=0$ for every sequence if the two models are identical and neither moves. Fix: ensure the policy starts from the SFT checkpoint while ${\pi }_{\text{ref}}$ is that same checkpoint but a genuinely frozen copy — with PEFT, the standard trick is to disable the adapter for the reference forward pass.

Forgetting that “the reward model” is now the policy. Symptom: you evaluate your implicit reward as if it were a calibrated RM and are surprised that it ranks unrelated pairs badly. Cause: ${\hat{r}}_{\theta }$ is only meaningful relative to ${\pi }_{\text{ref}}$ and only on the distribution the pairs came from. Fix: evaluate with held-out pairs, a judge, or a separate RM — not with ${\hat{r}}_{\theta }$ itself.

Assuming DPO is immune to reward hacking. Symptom: a good training loss and a model that games what the annotators (or the LLM labeller) liked. Cause: DPO optimises the same proxy, just offline. Fix: the same mitigations as RLHF — held-out evaluation, diverse pairs, human reading.

6. Using it in a project

Reach for DPO when you have offline preference pairs, no cheap environment, and no appetite for an RL loop. Style, tone, format, refusal behaviour, and single-turn preference alignment are its home territory: the desired behaviour is already present in the policy’s distribution (the SFT model can produce both the chosen and the rejected completion), and DPO only has to reweight the two. This is also why it is the standard second stage in the published pipeline: SFT → DPO → RLVR (Tulu 3, arXiv:2411.15124, states exactly that recipe).

Do not reach for DPO when:

• The reward is verifiable and the task is reasoning. DPO cannot sample, so it cannot discover a correct chain of thought that the SFT model rarely produced, and it has no group to compare. That is the GRPO/RLVR case. The evidence in 2026 is that online RL against a verifier is the path to reasoning capability; DPO remains the path to style/preference.
• Your preference data is thin. Use SFT, or collect more pairs. No $\beta$ fixes a signal made of a few hundred noisy comparisons.
• Your feedback is unpaired. If what you have is “this one was good” and “that one was bad”, that is KTO’s setting, not DPO’s.
• You need a reusable, stand-alone reward. DPO consumes the reward inside the policy; if you need to score candidates, run best-of-$n$, or drive a search, you want an explicit reward model (Reward Modeling).
• You can afford neither an RM nor sampling. This is the case DPO was designed for — and the honest framing is that DPO is the method you use when the RM path and the on-policy path are both closed to you, not when you have measured that it wins. Xu et al. 2024 finds PPO stronger in its alignment setting; Ahmadian et al. 2024 finds a simpler online method stronger still. DPO wins on cost and stability, and on offline data it is the only option.
• You need the policy to explore beyond its SFT distribution. DPO’s support is the pair dataset’s support.

The 2026 framing, stated plainly. [as of 2026-10] DPO is not legacy. It is still the default cheap offline alignment method, still in every framework (TRL, veRL, OpenRLHF), and still the warm start before RLVR. What changed is that it is no longer the destination: the frontier moved to online, critic-free, verifiable-reward RL for capability, and kept DPO for preference and style.

7. How you would prove it worked

Held-out pairwise accuracy, from a different source. Split the pairs; train on one part; report accuracy on the other. Then report a second number on pairs from a different dataset or a different annotator. (example) It is common for the first to reach 70%+ while the second barely moves — that gap is the overfitting signal, and it is why a single number is not evidence.

A win-rate against the SFT baseline, with a judge you did not train. Generate from both models on a held-out prompt set and have a judge (a strong model with a fixed rubric, or humans) pick a winner, blinded to which is which. This is the product-shaped metric; pairwise accuracy is the training-shaped one. Report both and the inter-rater agreement.

The likelihood decomposition. Log the mean $\log {\pi }_{\theta }(y_w\mid x)$ and $\log {\pi }_{\theta }(y_l\mid x)$ separately over training, alongside the gap. The symptom to catch is both falling — the model is winning the comparison by pushing the loser down rather than lifting the winner up. Without this plot, that failure looks like success.

The $\beta$ sweep. $\beta \in \{0.01,0.05,0.1,0.3\}$ (example) with everything else fixed, plotted as held-out pairwise accuracy against mean KL from the reference. The curve is the trade-off: too small drifts, too large does not move. Pick the point, not the extreme.

Length as a control. Report mean completion length for SFT and DPO outputs. If the win-rate gain disappears when you length-control (compare per-token, or truncate to matched length), you measured verbosity, not preference.

Reporting template (example). “DPO at $\beta =0.1$ for one epoch improved held-out pairwise accuracy from 62% to 69% and blinded win-rate against SFT from 48% to 61%. Mean completion length rose 7%; the per-token win-rate remained positive, so the gain is not purely verbosity. $\beta =0.01$ reached a higher training accuracy but its KL from the reference grew 6x and its blinded win-rate was no better.”

8. Interview drill

“Derive DPO.” Strong answer: four moves. (1) Start from the KL-regularised objective ${\max }_{\pi }\mathbb{E}[r]-\beta D_{KL}(\pi \Vert {\pi }_{\text{ref}})$. (2) Its optimum is the tilted policy ${\pi }_r\propto {\pi }_{\text{ref}}\exp (r/\beta )$, which rearranges to $r(x,y)=\beta \log ({\pi }_r/{\pi }_{\text{ref}})+\beta \log Z(x)$. (3) Replace ${\pi }_r$ with the parameterised ${\pi }_{\theta }$ to get the implicit reward ${\hat{r}}_{\theta }=\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$; the $\beta \log Z(x)$ term is dropped because Bradley–Terry uses reward differences for the same prompt, so it cancels. (4) Substitute into the preference likelihood and take the negative log: $-\log \sigma ({\hat{r}}_{\theta }(y_w)-{\hat{r}}_{\theta }(y_l))$. Common wrong answer: “DPO is just a ranking loss on log-probabilities.” It is a ranking loss on the reference-relative log-ratio, and that reference term is what makes it the RLHF objective rather than an ad-hoc pairwise hinge.

“What is the implicit reward, and why is it called a reward?” Strong answer: ${\hat{r}}_{\theta }(x,y)=\beta \log ({\pi }_{\theta }(y|x)/{\pi }_{\text{ref}}(y|x))$. It is called a reward because for the optimal policy it is the reward, up to a prompt-dependent constant — that is the closed-form inversion. It is not a calibrated reward model, though: it only means something relative to the reference and on the data distribution. Common wrong answer: “it is the reward model you no longer need.” It is a reward function parameterised by the policy; what you removed is the separate network and its training stage.

“Why does $\beta$ appear twice and is it the same $\beta$?” Strong answer: yes, one number, two readings — the KL penalty coefficient in the objective and the temperature in ${\pi }_r\propto {\pi }_{\text{ref}}\exp (r/\beta )$. In the loss it sets the sigmoid’s sharpness and the implicit reward’s units. It is not transferable to SimPO and friends, because they change the normalisation of the implicit reward. Common wrong answer: “$\beta$ is the learning rate.” It scales the update, but it is an algorithmic coefficient with a KL interpretation, not an optimiser setting.

“What does the DPO gradient actually do?” Strong answer: it increases the winner’s log-likelihood and decreases the loser’s, weighted by $\sigma ({\hat{r}}_{\theta }(y_l)-{\hat{r}}_{\theta }(y_w))$ — large when the model currently ranks the pair incorrectly, near zero when it already ranks it correctly. So the update is adaptive and the model stops spending gradient on pairs it has solved. Follow-up: “then why can both likelihoods fall?” → because only the difference is penalised; pushing the loser down can be cheaper than lifting the winner, and nothing pins the winner’s absolute probability.

“Does DPO beat PPO?” Strong answer: not in general, and the honest answer separates two settings. DPO is better when the data is offline, the budget is small, and behaviour is already in the policy’s support: cheaper, more stable, no reward model. PPO/online RL is better for reasoning and verifiable rewards, where you need the policy’s own samples and exploration. Xu et al. 2024 finds PPO stronger in their alignment study; Tulu 3 uses both, in that order. Common wrong answer: “DPO wins because it is simpler.” Simplicity is the cost argument, not the quality argument.

“Why can you skip the reward model but not the reference model?” Strong answer: the reward model is what the derivation eliminates — the implicit reward reproduces it. The reference is what makes the implicit reward well-defined: $\beta \log ({\pi }_{\theta }/{\pi }_{\text{ref}})$ measures divergence from ${\pi }_{\text{ref}}$, and with ${\pi }_{\text{ref}}={\pi }_{\theta }$ the reward is identically zero. Its log-probabilities are frozen, so they can be cached once. Common wrong answer: “DPO is reference-free.” That is ORPO and SimPO, not DPO.

“When would you choose DPO over GRPO in 2026?” Strong answer: when the task is preference/style/harmlessness rather than verifiable reasoning, when there is no cheap environment to sample from, and when the behaviour you want is already in the SFT model and merely needs reweighting. Choose GRPO when you have a verifier, can sample groups, and want capability. The published frontier recipe is SFT → DPO → RLVR: DPO for preference, RLVR for reasoning. Common wrong answer: “GRPO always now.” That ignores the entire offline preference workload and the fact that DPO needs no reward function at all.

Next: Monte Carlo, TD, and Search — the estimator family underneath PPO, GRPO, and every method in this part: returns versus bootstrapping, the bias–variance dial, and how a value estimate plus a policy prior becomes a search.