Reward Modeling

TL;DR — A reward model (RM) turns a finite set of human rankings into a scalar function that can be evaluated millions of times. It does so by assuming Bradley-Terry: the probability that $y_w$ beats $y_l$ is a logistic function of the difference of their latent scores, which makes maximum-likelihood fitting exactly a binary cross-entropy on pairs. Because the RM is fit to a finite sample from a distribution that the policy then moves, maximising it eventually stops improving the thing you actually care about: proxy reward keeps rising while gold reward falls. That is reward hacking, it is measured by Gao et al.’s scaling law, and it is detected by plotting proxy and gold against $d=\sqrt{D_{\mathrm{KL}}(\pi \Vert {\pi }_{\text{init}})}$ rather than against training step.

1. Why this exists

You have decided (see RLHF: The Three Stages) that the objective you want is a scalar preference score. Now try to write it down.

The naive move is to hand-write the reward: count the length, compute ROUGE against a reference, run a sentiment classifier, check the response for a forbidden word. Every one of these is a reward function, and every one of them fails the same way — the policy finds the shortest path to the number and ignores everything you meant. Reward length and you get padding until the context limit. Reward ROUGE and you get extractive copying that destroys reasoning. Reward sentiment and you get sycophancy.

The second naive move is to ask humans for scores. “Rate this response 1 to 7.” This fails for a different reason: humans are bad at absolute calibration and inconsistent between annotators, but they are fast and reliable at relative judgements. Ask the same person which of two answers is better, ten times, and you get a stable answer far more often than you get a stable Likert score.

So the input is comparisons, and the job of this chapter is the machinery that converts comparisons into something differentiable. Three constraints shape it:

1. The signal is relational, not absolute. You observe which of two responses was preferred, never the preference magnitude.
2. The signal is expensive. Each comparison costs human attention, so a single annotation must yield as many training signals as possible.
3. The output must be reusable and cheap. RL will call it millions of times, so it has to be a forward pass, not a human.

The structural point. A reward model is not “a model that scores text”. It is an amortisation: you spend an expensive, finite set of human judgements once, and you buy a cheap function you can query forever. Every property of the RM — including its hackability — follows from that trade. A function fit to a finite sample is only valid on the distribution that sample came from, and RL’s entire job is to move the distribution.

2. How it works

2.1 The Bradley-Terry model

Given a prompt $x$ and two completions $y_w$ (“winner”, the preferred one) and $y_l$ (“loser”), the model posits:

$$P(y_w\succ y_l\mid x)=\sigma (r(x,y_w)-r(x,y_l)),\sigma (z)=\frac{1}{1+e^{-z}}$$

Every symbol, and why it is where it is:

• $y_w\succ y_l$ — the event “a human prefers $y_w$ to $y_l$”. It is an event, not a score: this is the only thing the data contains.
• $r(x,y)\in \mathbb{R}$ — the latent score of completion $y$ for prompt $x$. It is latent because no human ever writes it down; it is what the RM is introduced to estimate. It is a function of the pair $(x,y)$, not of $y$ alone: the same response can be good for one prompt and bad for another, and a model that ignored $x$ would be fitting the marginal quality of text rather than its appropriateness.
• $r(x,y_w)-r(x,y_l)$ — the score difference. This is the only place $r$ enters. There is no separate “quality” term and no intercept. That single choice is what makes the model work with comparison data, and it has a hard consequence derived in §2.2.
• $\sigma$ — the logistic function, which squashes $\mathbb{R}\to (0,1)$ so the output is a probability. The choice is not arbitrary: it is the canonical link for a Bernoulli observation with a linear predictor, it is monotone (so the ordering of differences is preserved exactly), and — crucially — it makes the negative log-likelihood a softplus in the difference, which is smooth, convex in $r$ restricted to the pair, and has no vanishing-gradient plateau on the correct side.
• The absence of a temperature or scale parameter on the difference is a convention, not a modelling claim: $\sigma (\beta (r_w-r_l))$ is the same family for any $\beta >0$, and $\beta$ is absorbed into the scale of $r$. It reappears in DPO, where it is called $\beta$ and is not free — see DPO.

Why the difference, and what breaks if you drop it. If you instead modelled $P(\text{win}\mid y)=\sigma (r(x,y))$ — a pointwise model — you would be asserting that a response’s probability of winning is a property of that response alone. It is not: whether an answer is preferred depends on what it is being compared to. Concretely, a response that is merely adequate wins 100% of the time against garbage and 0% of the time against an excellent answer, yet a pointwise model must assign it one number. Pointwise models are what make DPO’s failures “pairwise-preference-inconsistent”; the generalisation is treated in IPO (Azar et al. 2023, arXiv:2310.12036).

2.2 The consequence: only differences are identified

Shift every score by an arbitrary per-prompt constant $c(x)$:

$$\sigma ((r_w+c(x))-(r_l+c(x)))=\sigma (r_w-r_l)$$

The likelihood is exactly unchanged. So $r$ is identified only up to a prompt-dependent additive constant, and any claim about the absolute level of a reward is meaningless. Two practical consequences, and they pull in opposite directions:

• Good news for the likelihood. The unidentified constant does not hurt training. There is no degeneracy to break and no need for a per-prompt normalisation inside the loss. This is why the reward trainer is a plain classification loss and not something cleverer.
• Bad news for the RL objective. The RLHF objective is $\mathbb{E}[r(x,y)]-\beta D_{\mathrm{KL}}$, and $r$ enters directly while the KL term is anchored at zero. Add a constant to every reward and you have changed the objective. So a reward whose origin is arbitrary must be re-centred before RL. InstructGPT: “since the RM loss is invariant to shifts in reward, we normalize the reward model using a bias so that the labeler demonstrations achieve a mean score of 0 before doing RL.” (Ouyang et al. 2022, arXiv:2203.02155).

This is the cleanest example in the whole part of a mathematical invariance that you must deliberately break in the pipeline.

2.3 The exact pairwise loss

Maximum likelihood under Bradley-Terry, as written in DPO’s paper (Rafailov et al. 2023, arXiv:2305.18290, Eq. 1):

$${\mathcal{L}}_R(r_{\varphi },\mathcal{D})=-{\mathbb{E}}_{(x,y_w,y_l)\sim \mathcal{D}}\left[\log \sigma \right(r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)\left)\right]$$

Symbol by symbol, with the “why it is there”:

• $r_{\varphi }$ — the reward model, parameters $\varphi$. Architecturally it is the SFT model with the language-modelling head replaced by a scalar head, so the reward is one number read off the final position. This reuse is why RM training costs about the same as fine-tuning, and why the RM inherits all the SFT model’s priors about what good text looks like.
• $\mathcal{D}$ — the comparison dataset: triples $(x,y_w,y_l)$.
• ${\mathbb{E}}_{(\cdot )\sim \mathcal{D}}$ — the empirical average over the dataset. In a batch implementation, this is loss.mean(), and the batch composition matters: pairs from one prompt are correlated, so a batch built by shuffling pairs puts near-duplicates in the same gradient.
• $r_{\varphi }(x,y_w)-r_{\varphi }(x,y_l)$ — the margin of the model for the preferred response. The loss pushes it up.
• $\log \sigma (\cdot )=-\log (1+e^{-\cdot })$ — the log-likelihood of the observed preference. Note $-\log \sigma (z)=\log (1+e^{-z})$, so the loss is a softplus and is never exactly zero: a pair can always be ranked more confidently, so the gradient never vanishes on the correct side. This is the same property that makes logistic regression keep moving, and it is why RM training “overfits” in a way that is hard to see in the loss curve.
• The leading minus converts log-likelihood into a loss to minimise.

Now read the loss as a classifier, because that is what it is. Define the binary label $z=1$ for “the first-listed response won”. Then the loss is exactly binary cross-entropy with logits equal to the reward margin. Three facts follow immediately and each is worth being able to produce on demand:

• If the RM is perfect, the loss still tries to go to $-\infty$, because $\log \sigma (z)\to 0$ only as $z\to \infty$. Practically this means the RM’s loss is not a stopping criterion and the RM’s scale is unbounded — one of the mechanisms behind over-optimisation (§2.6).
• Accuracy is the natural reporting metric, and implementations compute it as (rewards_chosen > rewards_rejected).mean(). Accuracy saturates fast; it is a weak signal after the first epoch.
• The margin is the second natural metric, and it is the one that keeps moving: (rewards_chosen - rewards_rejected).mean(). Watching the margin grow while held-out accuracy is flat is watching the RM become more confident, not better.

2.4 A runnable RM, from scratch

This is the same loss with nothing hidden. It runs on CPU with torch>=2.0, and it demonstrates the shift invariance of §2.2 and the margin term of §2.5.

"""A minimal Bradley-Terry reward model, from scratch.

torch >= 2.0. Illustrative: the "LM backbone" is a bag-of-token-ids scorer. The
loss, the invariance, and the margin behaviour are exactly those of a real RM.
"""
import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)
VOCAB = 16


class TinyRM(nn.Module):
    """[batch, seq] token ids -> [batch] scalar reward, read off the last step."""

    def __init__(self, vocab: int = VOCAB, dim: int = 32):
        super().__init__()
        self.embed = nn.Embedding(vocab, dim)
        self.trunk = nn.Sequential(nn.Linear(dim, dim), nn.Tanh())
        self.score = nn.Linear(dim, 1)          # the scalar head; num_labels=1

    def forward(self, ids: torch.Tensor) -> torch.Tensor:
        # Mean-pool here only because the backbone is a bag of ids. A real RM
        # reads the hidden state at the FINAL token, because that is the only
        # position whose representation has seen the whole sequence.
        h = self.trunk(self.embed(ids)).mean(dim=1)
        return self.score(h).squeeze(-1)


rm = TinyRM()
opt = torch.optim.AdamW(rm.parameters(), lr=1e-3)


def make_batch(n: int = 64, length: int = 8):
    """Synthetic pairs where the 'winner' has a higher token mean."""
    chosen = torch.randint(0, VOCAB, (n, length))
    rejected = torch.randint(0, VOCAB, (n, length))
    # Make winners systematic: shift token ids up, clip at the vocabulary edge.
    chosen = torch.clamp(chosen + 3, max=VOCAB - 1)
    return chosen, rejected


def bt_loss(r_chosen: torch.Tensor, r_rejected: torch.Tensor,
            margin: torch.Tensor | None = None) -> torch.Tensor:
    """L = -E[ log sigmoid( r(y_w) - r(y_l) - m ) ].  Exactly Eq. 1 (+ margin)."""
    diff = r_chosen - r_rejected
    if margin is not None:
        diff = diff - margin
    return -F.logsigmoid(diff).mean()


for step in range(400):
    chosen, rejected = make_batch()
    loss = bt_loss(rm(chosen), rm(rejected))
    opt.zero_grad()
    loss.backward()
    opt.step()
    if step % 100 == 0:
        with torch.no_grad():
            acc = (rm(chosen) > rm(rejected)).float().mean().item()
            marg = (rm(chosen) - rm(rejected)).mean().item()
        print(f"step {step:4d} loss {loss.item():.4f} acc {acc:.3f} margin {marg:+.3f}")

# --- the invariance: adding a CONSTANT to every reward changes nothing --------
with torch.no_grad():
    chosen, rejected = make_batch(256)
    before = bt_loss(rm(chosen), rm(rejected)).item()
    rm.score.bias.add_(100.0)                       # shift every reward by +100
    after = bt_loss(rm(chosen), rm(rejected)).item()
print(f"loss before shift {before:.6f}, after +100 shift {after:.6f}")   # identical

The final print is the point. A loss that is invariant to a +100 shift in the reward head’s bias is a loss that has told you nothing about where the reward should be centred — which is exactly why the pipeline centres it externally before RL begins.

2.5 The margin term: using graded annotations

If your annotators record how much better the winner was, the plain Bradley-Terry loss throws that information away. Llama 2 replaces the loss with a rating-aware margin (Touvron et al. 2023, arXiv:2307.09288, its Eq. 1 and Eq. 2):

$${\mathcal{L}}_{\text{ranking}}=-\log \sigma (r_{\varphi }(x,y_c)-r_{\varphi }(x,y_r)-m(r))$$

• $y_c$, $y_r$ — Llama 2’s names for chosen and rejected (the same objects as $y_w$, $y_l$; the notation differs between papers and it is worth normalising in your head).
• $r$ — the labeler’s 4-point rating of how much better the chosen response is. Note the symbol collision: $r$ is also the reward. Read the paper’s formulas with care.
• $m(r)$ — a discrete function of the rating: a larger required margin for “significantly better” pairs, a smaller one for “slightly better”.
• Why it helps: without it, all pairs are treated as equally informative, so a pair the annotator was barely sure about contributes the same gradient as a clear win. With it, the loss demands a bigger score gap exactly where a human saw one. Llama 2 reports improvements in RM accuracy “especially on samples where two responses are more separable”.

The design lesson generalises past Llama 2: if your annotations carry graded confidence, the loss should consume it. Throwing it away is free information discarded. The trainer support for this is a margin key in the data collator; see §3.1.

2.6 Why the RM is a proxy, and how it fails

The RM is an empirical estimate of a preference function, fit to a finite sample drawn from a distribution that the RL policy then changes. As the policy moves, the RM is increasingly evaluated off-distribution and its ranking becomes arbitrary. Maximise it hard enough and you are optimising the RM’s errors rather than human preference. This is Goodhart’s law, and it has a specific name in this literature: reward hacking or reward over-optimisation. The general framing predates LLMs (Amodei et al. 2016, arXiv:1606.06565 lists “reward hacking” as one of five concrete problems).

The measurement is Gao, Schulman & Hilton 2022, Scaling Laws for Reward Model Overoptimization, arXiv:2210.10760. The setup is the thing to remember: a gold RM plays the role of humans, a proxy RM is trained on labels the gold RM produces, the policy is optimised against the proxy, and the gold score is tracked throughout. Let

$$d:=\sqrt{D_{\mathrm{KL}}(\pi \Vert {\pi }_{\text{init}})}$$

• $D_{\mathrm{KL}}(\pi \Vert {\pi }_{\text{init}})$ — the KL divergence between the optimised policy and the policy it started from. It is the natural measure of “how far did you walk”.
• The square root is deliberate. The paper uses $d$ rather than the KL itself because the reward-KL relationship is quadratic in $d$; the observation that reward scales as $\sqrt{\text{KL}}$ is attributed to Bai et al. 2022.

Then the fitted laws are:

$$R_{\text{bon}}(d)=d({\alpha }_{\text{bon}}-{\beta }_{\text{bon}}d)$$

$$R_{\text{RL}}(d)=d({\alpha }_{\text{RL}}-{\beta }_{\text{RL}}\log d)$$

• $R$ — the gold reward of the optimised policy, with $R(0):=0$.
• $\alpha$ — the linear gain coefficient: the per-unit improvement from optimising the proxy at small $d$. This is genuine progress.
• $\beta$ — the over-optimisation coefficient: the rate at which the proxy diverges from the gold as you push. This is the Goodhart term.
• $d(\alpha -\beta d)$ — a concave parabola in $d$: rise then fall. The fall is the whole phenomenon. Gold reward is not a floor under the proxy; past the peak, more optimisation makes you worse.
• The two functional forms differ because best-of-$n$ and RL reach a given KL by different routes. For best-of-$n$ the paper gives the analytic relation ${\mathrm{KL}}_{\text{bon}}=\log n-(n-1)/n$, so a best-of-$n$ run can be placed on the same $d$ axis as an RL run.
• Both laws are functions of $d$, not of training step. That is why the x-axis of every diagnostic plot in this chapter is $d$. Training step is a proxy for compute; $d$ is the thing the law is actually about.

Four further verified findings, each of which changes a decision:

1. The coefficients scale smoothly with the number of proxy RM parameters, following approximate logarithmic trends, and ${\alpha }_{\text{RL}}$ is nearly independent of RM size. This is what makes the law predictive rather than descriptive: fit it at small scale, predict the peak at larger scale.
2. Policy size barely matters. Larger policies are better overall but over-optimise by a similar amount. Do not expect “a bigger actor” to fix hacking.
3. The KL penalty does not cure it. “Using a KL penalty increases the proxy reward model score that can be achieved for a given KL divergence, but this does not correspond to a measurable improvement in the gold RM score-KL frontier.” Read that carefully: the penalty still caps how far you drift — which caps the damage — but per unit of distance travelled it does not make the drift less harmful. The KL penalty is a distance limiter, not a Goodhart cure. The paper notes the result may be hyperparameter-sensitive.
4. Iteration helps, and by a knowable amount. With $k$ re-labelling rounds each covering distance $d/k$, the law becomes $R_{\text{RL}}(d)=d({\alpha }_{\text{RL}}-{\beta }_{\text{RL}}\log d+{\beta }_{\text{RL}}\log k)$. Iteration raises the final gold score by ${\beta }_{\text{RL}}\cdot d\cdot \log k$, and it does not repair the $\alpha$ term. This is the quantitative case for online/iterated RLHF, and it explains the annualised improvement curve: you get $\log k$ of it, so a 10x relabelling rate buys a fixed increment, not a cure.

2.7 Why the RM’s parameters are the same as the policy’s — and why that couples two decisions

InstructGPT is explicit: “we only use 6B RMs, as this saves a lot of compute, and we found that 175B RM training could be unstable and thus was less suitable to be used as the value function during RL.” Two distinct constraints are bundled in that sentence:

• RM quality improves with size, up to a point.
• Critic quality is coupled to RM quality, because in InstructGPT the value function is initialised from the RM. A 175B RM that trains unstably gives you a 175B critic that starts from garbage.

So the RM size is not a free hyperparameter; it is simultaneously a model-quality choice and a critic-initialisation choice. That is one more reason the critic-free methods in GRPO are attractive: they sever the coupling.

2.8 The thing that looks like a bug but is not

InstructGPT reports that “training for more epochs helps both the RM score and human preference ratings, despite this overfitting” — validation RM accuracy degrades while downstream human preference improves. The operational consequence is that “stop when validation accuracy peaks” is the wrong rule: the RM’s ranking accuracy measures fit to a fixed label distribution, which is not the same as how useful it is as an RL signal. RewardBench (Lambert et al. 2024, arXiv:2403.13787) is the standard standalone RM benchmark, but the honest evaluation is downstream.

3. Implementation

3.1 The real API: TRL RewardTrainer

The canonical implementation is RewardTrainer in TRL. The example below targets TRL v1.15.0 (released 2026-10-08) and transformers>=4.57; field names and defaults were read from trl/trainer/reward_config.py and trl/trainer/reward_trainer.py at that tag [as of 2026-10].

# pip install "trl==1.15.0" "transformers>=4.57" datasets accelerate
from datasets import load_dataset
from trl import RewardConfig, RewardTrainer

# 1. Data. Either a raw preference dataset with `chosen`/`rejected` columns, or
#    a pre-tokenised one with `chosen_ids`/`rejected_ids`. The 'prompt' column is
#    optional (implicit vs explicit prompt); an optional `margin` column is
#    consumed as the m in Eq. 2 of the Llama 2 ranking loss.
train = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
eval_ = load_dataset("trl-lib/ultrafeedback_binarized", split="test")

# 2. Config. Only the knobs that change behaviour are set explicitly.
config = RewardConfig(
    output_dir="rm-qwen25-05b",
    # --- model ---
    model_init_kwargs={"dtype": "bfloat16"},   # loads the BACKBONE in bf16
    disable_dropout=True,                      # default; see pitfall 4
    # --- data ---
    max_length=1024,        # DEFAULT 1024. Samples where either side exceeds
                            # this are FILTERED OUT, not truncated. That is a
                            # dataset-filtering decision disguised as a number.
    # --- optimisation ---
    learning_rate=1e-4,     # RewardConfig's default; 10x higher than the
                            # TrainingArguments default of 5e-5
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,   # effective batch 32
    num_train_epochs=1,
    gradient_checkpointing=True,     # default True in RewardConfig
    bf16=True,                       # default True when fp16 is unset
    logging_steps=10,                # default 10, not 500
    eval_strategy="steps",
    eval_steps=200,
    # --- the anti-drift regulariser, optional ---
    center_rewards_coefficient=0.01,  # DEFAULT None. Adds
                                      # c * mean((r_chosen + r_rejected)^2), which
                                      # pushes the mean reward toward zero so the
                                      # RL KL term is not swamped by an offset.
)

trainer = RewardTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",   # a CAUSAL LM checkpoint, not a
                                          # sequence-classification one: the
                                          # trainer loads it with
                                          # AutoModelForSequenceClassification
                                          # and num_labels=1
    args=config,
    train_dataset=train,
    eval_dataset=eval_,
)

trainer.train()
trainer.save_model("rm-qwen25-05b")   # saves the backbone + the scalar head

Five facts about that code that a reader cannot guess and that matter:

• model is a causal-LM checkpoint. The trainer re-loads it through AutoModelForSequenceClassification with num_labels=1, which discards the LM head and adds a randomly initialised scalar head. You will see a warning about the missing score.weight key; it is expected.
• max_length filters, it does not truncate. At the default of 1024, long comparison pairs are silently dropped. On a long-form preference dataset that can remove a large fraction of the data, and the surviving distribution is biased toward short responses — which biases the RM, which biases the policy. If you see a suspiciously short mean response length after RL, check this number first.
• center_rewards_coefficient is the pre-RL bias normalisation, moved into the loss. Instead of post-hoc shifting the reward head, it adds a penalty that encourages mean-zero rewards. Either mechanism works; using neither is the mistake.
• The trainer logs accuracy and margin over the eval set, and also min/mean/max_reward. Track accuracy for whether the RM ranks correctly and margin for whether it is becoming overconfident. A run where accuracy is flat and margin is growing is a run entering the danger zone of §2.6.
• gradient_checkpointing defaults to True, which is unusual for a TrainingArguments subclass and is there because RM sequences are long.

3.2 What the trainer does not do

RewardTrainer implements only the pairwise Bradley-Terry loss (plus the optional margin and centring terms). It does not provide:

• Ensembling. You train $N$ RMs yourself and average their scores (or their weights — see §3.3).
• The over-optimisation tripwires. You write the proxy-vs-gold plot and the $d$-vs-step plot.
• The reward normalisation for RL. The center_rewards_coefficient term is a training-time incentive, not a guarantee; the RL stack still does reward whitening per minibatch.
• Any guarantee the RM is trained on the right distribution. That is a data decision you make.

3.3 Ensembling, and the honest version of it

If one RM is one over-fittable function, averaging several reduces the exploitable surface. Two mechanisms, and they are not equivalent:

• Inference-time ensembling — average the scores of $N$ independently trained RMs. Costs $N$ forward passes per rollout.
• Weight averaging (WARM) — average the weights of independently fine-tuned RMs, producing one model. Costs nothing extra at rollout time (Ramé et al. 2024, arXiv:2401.12187).

Coste et al. 2023, arXiv:2310.02743 is the direct study. The paper most worth knowing here is Eisenstein et al. 2023, Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking, arXiv:2312.09244, because its title is the finding: ensembles that vary by pretraining seed generalise better than ensembles that differ only by fine-tuning seed — and even those do not eliminate hacking, because all members of an ensemble trained the same way share the same error patterns. The ensemble disagreement is nevertheless a useful uncertainty probe: where members disagree is where the proxy is least trustworthy.

Do not attribute ensembling to Llama 2. Its RM section contains no “ensemble” language; it trains separate helpfulness and safety RMs, which is a different design. Coste et al. and WARM are the right citations.

4. Numbers and trade-offs

4.1 The cost of an RM, and what it buys

The economically interesting column is the second: the RM is how a fixed annotation budget becomes an unbounded number of reward queries. If you paid a human $1 per comparison and bought 10k comparisons, you have spent $10k once and can now score every rollout for the rest of the run. That is the whole value proposition, and it is also the whole risk: the reuse is what lets the policy over-fit the RM.

4.2 The trade-off curve you are actually choosing on

Three axes, and every RM decision moves you along one of them:

The over-optimisation coefficients scale roughly logarithmically with RM parameter count, which means the returns to a bigger RM are real but diminishing — the same shape as everything else in this book.

4.3 A worked (example) budget

(example) Suppose 20k prompts, each with $K=4$ ranked completions, and a mean completion length of 300 tokens. That is 80k completions and $20,000\times \left(\genfrac{}{}{0ex}{}{4}{2}\right)=120,000$ pairs. Tokenising both sides gives $120,000\times 2\times 300=7.2\times 10^7$ tokens per epoch — about the size of a medium SFT run, and about $\left(\genfrac{}{}{0ex}{}{K}{2}\right)/(K-1)=K/2=2$ times what a top-1-vs-rest scheme would consume on the same annotations ($120,000$ pairs against $60,000$). If the RM is the same size as the policy you are aligning, that is one extra fine-tuning run in your pipeline for the privilege of an unbounded reward function.

5. Pitfalls

Optimising the proxy without a gold signal. You have an RM, you run RL against it, and nothing tells you when the reward stopped meaning anything. Symptom: the training reward curve climbs monotonically to the end of the run and looks like the healthiest curve in the project. It is the most dangerous curve in the project. The fix is organisational, not algorithmic: a frozen, held-out preference signal that is never used for training or checkpoint selection.

Stopping RM training when validation accuracy peaks. Symptom: you select the epoch with the best held-out ranking accuracy and get a worse RL run than a colleague who trained longer. InstructGPT reports exactly the counter-intuitive behaviour that causes this: more RM epochs improve downstream human preference while validation accuracy degrades. Select on downstream preference, or at least refuse to prune on validation accuracy alone.

Adding a constant to the reward and expecting the same RL run. Symptom: the KL term consumes the entire reward at the start; the policy is pulled back to the reference and the reward curve is flat while kl is large. The cause is §2.2: the likelihood is shift-invariant but the RL objective is not. Centre the reward against the SFT responses, or use center_rewards_coefficient.

max_length truncating/length-filtering your dataset. Symptom: after RL, mean response length falls, and long-form performance degrades. In TRL, pairs longer than max_length (default 1024) are filtered, so the RM never sees long responses and learns a length-correlated preference. Either raise the limit or verify explicitly that your retained data is not length-biased.

Dropout on the reward head. Symptom: RM scores are noisy at evaluation, because a stochastic forward pass gives a stochastic reward; and RL trains against a moving reward function. RewardConfig defaults disable_dropout=True for exactly this reason. Do not turn it back on.

Batching correlated pairs. Symptom: gradient noise looks fine at batch size 64, but the RM generalises worse than the identical run with a shuffled, prompt-disjoint batch split. All $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs from one prompt share a prompt; keeping them in one batch reduces the effective sample size by roughly $K$.

Treating RM accuracy as the delivery metric. Symptom: the RM wins your internal ranking benchmark and the RL’d policy is still worse than SFT. RM benchmarks measure fit to a label distribution; the deliverable is a policy. Report downstream win-rate, and report the RM’s accuracy only as a sanity check.

Assuming the reward model removes the reward-specification problem. Symptom: the model becomes sycophantic, verbose, or pathologically hedged — the exact failure modes of a hand-written reward — but now you cannot read the reward function to find out why. An RM is a learned reward function; it inherits every problem of a hand-written one and adds an estimation step.

Annotator drift. Symptom: the RM’s preferences change character mid-dataset and the loss has an unexplained plateau. Preference data collected over weeks, by changing pools, with changing guidelines, is not i.i.d. Version your instructions alongside your data.

6. Using it in a project

6.1 When a learned RM is the right tool

• The objective is real but not checkable. Tone, helpfulness, harmlessness, clarity, “would a user trust this answer”. No string comparison can score it.
• You have (or can buy) thousands of comparisons. Below a few thousand pairs the RM is fitting noise, and the policy will optimise the noise.
• You will call the reward many times. The point of an RM is amortisation. If you only score a few thousand completions, pay a human or a model judge directly and skip the RM entirely.
• The reward is not a single scalar you can compute. If it is, use RLVR.

6.2 When NOT to use a learned RM

• When the task is verifiable. Math answer match, unit tests, schema validation, a compiler. Use RLVR: the signal is cheap, deterministic, and unfakeable with respect to the checked property. Building an RM to learn something a string comparison already knows is the most common wasteful choice in this part. Tulu 3 introduces the term and the recipe (Lambert et al. 2024, arXiv:2411.15124).
• When your preference data is thin. A few hundred pairs produces an RM that is confidently wrong in ways the policy finds immediately.
• When you cannot afford a frozen held-out preference signal. Without one, you cannot distinguish an improvement from over-optimisation, and selecting on the proxy is selecting for the hack.
• When the difference between good and bad is a policy you can express. “Never echo the system prompt” is a filter. “Always cite the source” is a prompt instruction. Do not train a reward model to enforce something a conditional can enforce.
• When a model judge is good enough at inference time. Reranking $n$ candidate responses with an off-the-shelf strong model costs no training run. It is not as cheap per query, but if your total query volume is small it wins outright.

6.3 The decision tree, condensed

1. Is the desired property checkable? Yes → RLVR, stop here.
2. Do you have pairs already, and no environment to sample from? Yes → DPO or SimPO on those pairs (DPO), stop here. An RM is not needed; DPO’s policy is an implicit RM.
3. Can you sample and score with a program? Yes → RLVR with GRPO.
4. Otherwise → train an RM, and budget for the gold signal at the same time you budget for the annotations. If you cannot afford both, you cannot afford the RM.

7. How you would prove it worked

7.1 The three plots

The RM’s own metrics are not the deliverable; these are.

1. Proxy reward and gold reward on one axis, against $d$ (not against step). Gao et al.’s laws are functions of $d$, so $d$ is the correct x-axis. The diagnostic is the gap and the sign of the gold curve’s slope. Gold flat or falling while proxy rises means you are past the peak of $d(\alpha -\beta \log d)$.
2. $d=\sqrt{D_{\mathrm{KL}}(\pi \Vert {\pi }_{\text{init}})}$ against training step. This tells you how fast you are spending your over-optimisation budget. If $d$ grows much faster than intended, $\beta$ is too small or the reward scale is too large.
3. Held-out pairwise accuracy of the RM, evaluated on the current policy’s outputs. A rising proxy reward with falling pairwise accuracy on freshly sampled policy outputs is the classic hacking signature: the policy has found the region where the RM is wrong.

7.2 The ablation that isolates the RM

Run identical RL from an identical SFT checkpoint against:

1. Ground-truth / gold reward (human or a strong frozen model judge). This is the ceiling for what the algorithm could achieve.
2. The trained RM. This is what you ship.
3. A deliberately weak RM (smaller, or trained on 10% of the pairs). This measures the imputation gap: how much of the loss comes from the RM rather than from RL.

(example) A typical small-scale result is that arms 1 and 2 track each other up to some $d$, then diverge, with the divergence point moving earlier as the RM gets weaker. That divergence point is your usable $d$ budget, and finding it is worth more than any single hyperparameter sweep.

7.3 A small but high-value check

Score the SFT model’s own outputs (the training distribution) with the RM and verify the mean is near zero before RL starts. If it is not, you have skipped the bias normalisation, and your first hundred RL steps will be spent fighting a constant offset rather than learning a preference. This is a two-line check that saves runs.

8. Interview drill

“Derive the reward-model loss from the preference assumption.”

Strong answer: assume Bradley-Terry, $P(y_w\succ y_l\mid x)=\sigma (r_w-r_l)$. Take the log-likelihood of the observed preferences, negate it, and average over the dataset: $\mathcal{L}=-\mathbb{E}[\log \sigma (r_w-r_l)]$, which is binary cross-entropy with the reward margin as the logit. Follow-up: “what does that make the RM architecturally?” The SFT model with the LM head replaced by a scalar head, so only the final-token representation matters.

Common wrong answer: “it’s a regression on human scores.” That is the pointwise formulation, and it is the one Bradley-Terry exists to avoid.

“Why only the difference of rewards?”

Strong answer: because that is all the data identifies. A $K$-way ranking is a set of pairwise comparisons, and each comparison tells you only which response is better. Shift every reward by a prompt-dependent constant and the likelihood is unchanged, so the reward is identified only up to that constant. Follow-up: “so what breaks?” The RL objective, because there $r$ enters additively while the KL term is anchored at zero. Hence reward bias normalisation before RL — you deliberately break an invariance that training cannot see.

“The RM’s validation accuracy peaked at epoch 2. Do you stop?”

Strong answer: not on that alone. InstructGPT reports that training the RM for more epochs improves downstream human preference ratings despite overfitting on the RM’s own validation metric. The RM’s ranking accuracy measures fit to a fixed label distribution; what you need is a policy that wins on held-out preference. Select on the downstream signal.

Common wrong answer: “yes, that’s standard early stopping.” Why wrong: the metric and the deliverable are different objects, and this is the one documented case where they disagree in a specific direction.

“How do you detect reward hacking?”

Strong answer: plot gold/held-out reward against proxy reward on the $d$ axis and watch the gap; keep a frozen eval set that is never trained on or selected on. Then add cheap tripwires: response length (the canary), format and refusal statistics, n-gram entropy, and ensemble disagreement. Finally, read the outputs — reward hacking in RLHF is usually legible to a human as sycophancy, padding, or obsequiousness.

Follow-up: “does a bigger KL penalty fix it?” No. Gao et al. find the KL penalty raises the achievable proxy score at a fixed KL but does not improve the gold score-KL frontier. It limits distance, not damage per unit distance.

“Would an ensemble of reward models solve this?”

Strong answer: it reduces but does not eliminate it. Eisenstein et al. show that ensembles varying by pretraining seed generalise better than ones differing only by fine-tuning seed, and that even pretrain-seeded ensembles fail on hacking behaviours where all members share the same error pattern. Ensembling also gives you a disagreement signal, which is a cheap uncertainty estimate.

Wrong answer to avoid: attributing ensembling to Llama 2. It trains separate helpfulness and safety RMs; that is not an ensemble.

“You have 500 preference pairs and a 7B model. Now what?”

Strong answer: do not train an RM. With that much data the RM fits the annotators’ idiosyncrasies, and RL will optimise exactly those. Options in order of preference: check whether the task is actually verifiable and use RLVR; if not, spend the budget on more comparisons rather than on RL; or use the pairs for a short DPO run, which at least cannot explore off-distribution the way RL does.

Common wrong answer: “train the RM on the 500 pairs and use a strong KL penalty.” A strong KL penalty means you barely leave the SFT model, at which point you have paid for an RL pipeline and received an SFT model.

“What is the difference between the RM and DPO’s implicit reward?”

Strong answer: DPO’s implicit reward is ${\hat{r}}_{\theta }(x,y)=\beta \log ({\pi }_{\theta }(y\mid x)/{\pi }_{\text{ref}}(y\mid x))$ — the policy is the reward model, and the reference model supplies the origin. Consequences: no separate RM to train or serve, but also no way to sample and rescore independently of the policy, and a reward whose scale is tied to $\beta$. (Forward reference: the derivation is in DPO.)

“Name a failure of the RM that is not over-optimisation.”

Strong answer: the reward is read off the final token, so the RM’s judgement is a function of sequence position and length. Train on a length distribution and score off it and the reward is systematically wrong in a way that has nothing to do with preference. This is one of the mechanisms behind the length drift that SimPO calls out as its motivation (arXiv:2405.14734).

Next: PPO for LLMs — the reward model gives you a scalar per completion; PPO is the machinery that turns that scalar into a bounded policy update without letting the ratio explode.