PPO for LLMs

TL;DR — PPO is what made RLHF work, and its trick is refusing to let one batch of data move the policy too far. It reuses each rollout for several gradient epochs, corrects the resulting distribution mismatch with the probability ratio $r_t(\theta )$, then clips that ratio to $[1-ϵ,1+ϵ]$. Because the objective takes a min over the clipped and unclipped terms, moving outside the trust region is never rewarded, only penalised. The advantage ${\hat{A}}_t$ — in LLM RL computed by GAE through a learned critic — says which tokens to make more likely, and a per-token KL penalty to the frozen SFT reference keeps the policy inside the reward model’s region of validity. That is four models in memory at once (actor, critic, reward, reference), which is exactly the cost GRPO removes. [as of 2026-10] TRL has deleted its PPO trainer; OpenRLHF, veRL, DeepSpeed-Chat and SkyRL still implement PPO.

1. Why this exists

Start with the naive thing, because PPO is a response to three specific ways it breaks. You have a reward model $r_{\varphi }(x,y)$ (see Reward Modeling) and you want to maximise it:

$$J(\theta )={\mathbb{E}}_{x\sim \mathcal{D},y\sim {\pi }_{\theta }(\cdot \mid x)}[r_{\varphi }(x,y)]$$

The REINFORCE / vanilla policy-gradient estimate of its gradient is

$${\nabla }_{\theta }J(\theta )=\mathbb{E}[r_{\varphi }(x,y){\nabla }_{\theta }\log {\pi }_{\theta }(y\mid x)]$$

This is unbiased and it is nearly unusable for LLM post-training. Three failures:

It throws away every sample after one step. Generating a 4,000-token response is a full autoregressive decode; scoring it is one forward pass. A single gradient step per generated batch means you pay the expensive half once and use it once. The obvious fix — take several gradient steps on the same batch — breaks the estimator, because after the first step the data no longer comes from ${\pi }_{\theta }$. That mismatch is exactly what the ratio in PPO repairs.

The step size is unbounded. The gradient says “make this completion more likely” with a magnitude proportional to the reward. Nothing stops one large step from moving the policy far off the distribution the reward model was trained on. The RM was fit on SFT-policy outputs; off that distribution its scores are arbitrary, so the policy walks into a region where a large RM score means nothing. Gao, Schulman & Hilton 2022 (arXiv:2210.10760) measured this as a scaling law in $d=\sqrt{D_{KL}({\pi }_{\theta }\Vert {\pi }_{\text{init}})}$:

$$R_{RL}(d)=d({\alpha }_{RL}-{\beta }_{RL}\log d)$$

Read the shape, because it is the whole motivation for a trust region. At $d=0$ the reward is zero; it rises linearly at first; then the $\log d$ term turns it over and the gold reward falls while the proxy RM reward keeps rising. $d$ is the square root of the KL divergence from the starting policy — the paper uses $\sqrt{KL}$ rather than $KL$ precisely because the relationship is quadratic in that variable. ${\alpha }_{RL}$ and ${\beta }_{RL}$ are positive fitted coefficients that scale roughly logarithmically with RM size. If you have never seen that curve: reward goes up, then down, and the down-slope is not a bug you can prompt your way out of. You have to bound the distance.

The raw return is a terrible advantage. Using $r_{\varphi }(x,y)$ directly as the per-token weight means every token of a good response is pushed up equally hard, regardless of whether it was the token that mattered, and the variance of the estimator is the variance of the reward across samples — large. Subtracting a baseline fixes the variance without introducing bias, because ${\mathbb{E}}_{a\sim \pi }[A^{\pi }(s,a)]=0$. The baseline for PPO is a learned value function, and sourcing that value function is the single largest engineering cost in the algorithm.

The one-sentence framing. PPO is “take several gradient steps on the same rollout, but clip the ratio so no single step is allowed to move the policy outside a small neighbourhood of the policy that generated the data.”

PPO was published for continuous control (Schulman et al. 2017, arXiv:1707.06347); almost none of the LLM-specific work — the critic on the LM backbone, the terminal reward, the KL folded into the reward — is in the paper. That adaptation is where the bugs live.

2. How it works

2.1 The probability ratio

Take a batch of prompts, sample completions from the current policy, then freeze a copy of the weights as ${\pi }_{{\theta }_{\text{old}}}$. Now every gradient step in this cycle is off-policy with respect to the data, and importance sampling is the correction:

$$r_t(\theta )=\frac{{\pi }_{\theta }(a_t\mid s_t)}{{\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)}$$

• $s_t$ — the state: in LLM terms, the prompt plus the first $t-1$ generated tokens. There is no hidden environment state; the transition is “append the token”, deterministic, and all randomness lives in the policy.
• $a_t$ — the action: the $t$-th generated token. The action space is the vocabulary.
• ${\pi }_{\theta }(a_t\mid s_t)$ — the current policy’s probability of that token, differentiated through the backward pass.
• ${\pi }_{{\theta }_{\text{old}}}(a_t\mid s_t)$ — the probability the sampling policy assigned, stored from the rollout and treated as a constant. Snapshot the weights (or, better, the log-probabilities) at the top of each cycle.
• $r_t(\theta )$ — the ratio. It is $1$ at the snapshot by construction. $r_t>1$ means the token has become more likely than when it was sampled; $r_t<1$ means less likely. This is the importance-sampling weight that makes multi-epoch updates on one batch a valid estimator. Remove it and epochs two through $K$ are gradients of a surrogate for a policy that no longer exists — the update is silently biased in the direction the first epoch already moved.

Why a ratio and not the log-probability? Because the surrogate is ${\mathbb{E}}_{a\sim {\pi }_{{\theta }_{\text{old}}}}[r_t(\theta ){\hat{A}}_t]$, and $r_t$ is the density ratio that converts an expectation under the old policy into one under the new. The gradient of $r_t{\hat{A}}_t$ with respect to $\theta$ is ${\hat{A}}_t{\nabla }_{\theta }\log {\pi }_{\theta }(a_t|s_t)$ times $r_t$, which is REINFORCE with the correct correction.

2.2 The clipped surrogate, term by term

This is PPO’s actual contribution. The paper’s Equations 6 and 7:

$$L^{\text{CLIP}}(\theta )={\hat{\mathbb{E}}}_t\left[\min \right(r_t(\theta ){\hat{A}}_t,\mathrm{clip}(r_t(\theta ),1-ϵ,1+ϵ){\hat{A}}_t\left)\right]$$

• ${\hat{A}}_t$ — the estimated advantage: how much better taking token $a_t$ was than the policy’s own average behaviour at $s_t$. ${\hat{A}}_t>0$ means “make this token more likely”; ${\hat{A}}_t<0$ means “make it less likely”. All credit assignment — GAE, baselines, group means — is squeezed into this one scalar per token. Remove it and the objective becomes $\min (r_t,\mathrm{clip}(r_t))$, which for the positive-ratio case is just $r_t$: the policy would push every sampled token up equally and collapse onto its own samples. It is the only term that distinguishes good tokens from bad ones.
• $ϵ$ — the clip range, a hyperparameter. The paper writes “say, $ϵ=0.2$”; the N+ RLHF reproduction verifies $ϵ=0.2$ (arXiv:2403.17031). It defines the trust region: the ratio is allowed to move at most a factor $1+ϵ$ up or $1-ϵ$ down before the gradient is switched off. Larger $ϵ$ → faster but less stable learning; smaller $ϵ$ → more conservative, more epochs needed. Removing it (equivalently, $ϵ\to \infty$) gives the unclipped $r_t{\hat{A}}_t$ and the policy can jump arbitrarily far in one update.
• $\mathrm{clip}(r_t,1-ϵ,1+ϵ)$ — clamps the ratio into the interval, which makes the second term constant with respect to $\theta$ whenever $r_t$ is outside it. A constant contributes zero gradient, so once you have already taken a step and are pushing further, you earn nothing more.
• $\min (\cdot ,\cdot )$ — the term that makes the objective pessimistic. It is the subtlest and most-asked part, so take it case by case.

Why the min makes this a lower bound. Write out the regions, splitting each advantage sign into the in-range case and the two out-of-range cases. Let $\hat{A}>0$ (a good token) first:

The two rows that say “equal” are equal only inside the trust region $[1-ϵ,1+ϵ]$; outside it on the unclipped side the two terms differ, but the min still selects the unclipped $r_t\hat{A}$, so the gradient keeps flowing. The pattern: whenever the policy has already moved in the direction the advantage wants and is now trying to move further outside the trust region, the objective is flat. Inside the region, the gradient is the ordinary policy gradient. Therefore the min never rewards leaving the trust region in either direction — it only removes the incentive to. In the second row, for instance, a good token can be made at most $(1+ϵ)$ times more likely before PPO stops paying for it; in the sixth row, a bad token’s probability can be pushed down to at most $1-ϵ$ times its sampling value before PPO stops paying for that too. That asymmetry — it caps the upside and the downside but never the punishment for going too far — is what makes the objective a lower bound on the unclipped surrogate $r_t{\hat{A}}_t$, which is itself the first-order approximation to the improvement in true return. Maximising a lower bound is safe; maximising the raw surrogate is where the policy explodes.

The connection to TRPO. Trust Region Policy Optimization (Schulman et al. 2015, arXiv:1502.05477) constrains the update by an explicit KL constraint with a second-order solve (conjugate gradients, Fisher-vector products). PPO’s clip is a cheap first-order stand-in for the same idea. The PPO paper’s own ablation, verified from its §6.1, is that clipping beat both an unclipped objective and a fixed/adaptive KL penalty in its setting — simpler and better. Keep that in mind: the KL term in LLM PPO is not the same device as TRPO’s constraint; it is a reward shaping term on top of clipping.

2.3 The advantage: GAE, and the dial that sets its bias-variance position

The advantage is defined as

$$A^{\pi }(s,a)=Q^{\pi }(s,a)-V^{\pi }(s)$$

• $Q^{\pi }(s,a)$ — the expected return after taking $a$ in $s$ and then following $\pi$.
• $V^{\pi }(s)$ — the expected return from $s$ under $\pi$, i.e. the baseline.
• The identity that makes baselines legal: ${\mathbb{E}}_{a\sim \pi }[A^{\pi }(s,a)]=0$. Subtracting any function of the state leaves the expected policy gradient unchanged but can shrink its variance enormously.

We do not have $A^{\pi }$, so we estimate it. Generalized Advantage Estimation (Schulman et al. 2015, arXiv:1506.02438) starts from the one-step TD residual:

$${\delta }_t^V=r_t+\gamma V(s_{t+1})-V(s_t)$$

• $r_t$ — the reward at step $t$. In the LLM adaptation this is normally $0$ for every non-final token and the reward-model score at the EOS token, plus the per-token KL penalty (Section 2.4). That the reward is terminal and sparse is the defining property of this MDP, and it is why the critic is hard to fit.
• $V(s_t)$ — the critic’s value estimate, a scalar produced by the same backbone as the actor with a scalar head.
• $\gamma$ — the discount factor. Its job is horizon control: how much future reward counts. $r_t+\gamma V(s_{t+1})$ is the bootstrap estimate of the return; subtracting $V(s_t)$ gives the “surprise” — how much better than expected this step turned out. If $V$ is wrong, the surprise is wrong, and that error is the source of GAE’s bias.
• ${\delta }_t^V$ — the TD residual.

GAE then takes an exponentially-weighted average over all $k$-step estimators, which collapses to a single recursion:

$${\hat{A}}_t^{\text{GAE}(\gamma ,\lambda )}=\sum _{l=0}^{\infty }(\gamma \lambda {)}^l{\delta }_{t+l}^V$$

• $\lambda \in [0,1]$ — the GAE decay. It is a bias-variance dial that does not change the effective horizon, because it always appears as the product $\gamma \lambda$: $\gamma$ sets the problem’s horizon, $\lambda$ sets the estimator’s position. Standard practice in LLM RL is $\lambda =0.95$; the N+ reproduction verifies exactly that.
• $(\gamma \lambda {)}^l$ — the weight on the residual $l$ steps ahead. The equivalent telescoping form in the paper is $(1-\lambda )({\hat{A}}_t^{(1)}+\lambda {\hat{A}}_t^{(2)}+{\lambda }^2{\hat{A}}_t^{(3)}+\dots )$; the $(1-\lambda )$ is what makes the weights sum to $1$ so that $\hat{A}$ is a convex combination of $k$-step estimators rather than an unnormalised sum.

The two degenerate settings, and what each does. These are the interview questions:

• $\lambda =0$: ${\hat{A}}_t={\delta }_t^V=r_t+\gamma V(s_{t+1})-V(s_t)$. “Trust the critic completely; react to one step.” Low variance, but biased whenever $V$ is not the true value function — and in LLM RL $V$ is bad early, because the value head is freshly initialised and the reward only arrives at EOS. In the worst case the advantage for every non-final token is $V(s_{t+1})-V(s_t)$, a difference of two untrained numbers.
• $\lambda =1$: ${\hat{A}}_t={\sum }_l{\gamma }^l{\delta }_{t+l}={\sum }_l{\gamma }^l{r}_{t+l}-V(s_t)$. “Ignore the critic except as a baseline; use the empirical return.” This is $\gamma$-just (unbiased for the true advantage up to the baseline) but has the variance of a sum of $T-t$ random terms. With $\gamma =1$ and a terminal reward it is literally “the sequence reward minus the critic’s baseline”.
• $0<\lambda <1$: the paper’s own words are “a compromise between bias and variance, controlled by parameter $\lambda$”.

Why $\gamma =1.0$ in LLM RL. A response is a bounded episode and the reward arrives at the end of it. Discounting would make the objective depend on where the answer happened to stop being generated, and would systematically discount the very signal you are training on. So $\gamma =1$ and the entire bias-variance burden sits on $\lambda$ and on the critic’s quality. The N+ reproduction verifies $\gamma =1.0$, $\lambda =0.95$.

In practice implementations backward-scan the token sequence: $(\gamma \lambda )$ is a constant, so the recursion ${\hat{A}}_t={\delta }_t+\gamma \lambda {\hat{A}}_{t+1}$ is one pass. That is why GAE costs nothing beyond the critic forward pass.

Bandit caveat. InstructGPT frames RLHF as a bandit: “The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt… it produces a reward determined by the reward model and ends the episode” (Ouyang et al. 2022, arXiv:2203.02155, §3.5). In a strict bandit there is no future state to bootstrap from. Implementations nevertheless keep per-token GAE with $\gamma =1$, $\lambda =0.95$, which is a within-response bootstrap using the value head as a per-token return predictor. Both framings are true; the bandit framing explains why the reward is terminal, the GAE framing explains what the code actually computes.

2.4 The KL penalty, and why the reference model is mandatory

The policy has a second leash besides the clip: a per-token penalty against a frozen reference policy ${\pi }_{\text{ref}}$, almost always the SFT checkpoint. InstructGPT: “we add a per-token KL penalty from the SFT model at each token to mitigate over-optimization of the reward model.” The penalty is folded into the reward rather than into the loss, so the RL machinery sees one scalar per step:

$$r_t^{\text{total}}=r_t^{RM}\cdot 1[t=T]-\beta (\log {\pi }_{\theta }(a_t\mid s_t)-\log {\pi }_{\text{ref}}(a_t\mid s_t))$$

• $r_t^{RM}\cdot 1[t=T]$ — the sequence reward, applied only at the last token. $1[t=T]$ is the indicator that this is the terminal step. Every token gets the same sequence-level reward through the bootstrap, but only EOS carries it directly.
• $\beta$ — the KL coefficient: how many units of reward one nat of divergence from the reference costs. InstructGPT verifies $\beta =0.02$; the N+ reproduction uses $\beta =0.05$. Those differ by more than 2x and both are correct, because $\beta$ is entangled with the reward scale and the normalisation scheme — never compare $\beta$ across codebases.
• $\log {\pi }_{\theta }-\log {\pi }_{\text{ref}}$ — the per-token log-ratio. In expectation over $a_t\sim {\pi }_{\theta }$ this is $D_{KL}({\pi }_{\theta }(\cdot |s_t)\Vert {\pi }_{\text{ref}}(\cdot |s_t))$, but the single-sample form is an unbiased yet high-variance estimate of it. This is why $\beta$ often has to be small: a noisy penalty with a large coefficient injects noise into the reward signal, not just a leash. The lower-variance alternative is Schulman’s $k_3$ estimator (joschu.net/blog/kl-approx.html), $k_3=\rho -\log \rho -1$ with $\rho ={\pi }_{\text{ref}}/{\pi }_{\theta }$, which is non-negative and drops the high-variance term.

Three reasons the KL term is not optional, each with the failure it prevents:

1. It bounds the reward-hacking distance. Gao et al.’s damage is a function of $d=\sqrt{KL}$; with no penalty the policy walks out to large $d$ quickly and the gold reward falls. The theoretical backing is the trust-region monotonic-improvement bound — how much the true return can differ from the estimate made under the old policy, as a function of the KL between them. Note the sharp caveat from that same paper: the KL penalty raises the proxy score achievable at a given $d$ without improving the gold-score-versus-$d$ frontier. The KL penalty is a distance limiter, not a cure for Goodhart.
2. It preserves language quality. The reward model was trained on the SFT policy’s distribution. Off that distribution its judgement is arbitrary, and unconstrained RM maximisation produces degenerate text — repeated tokens, format soup, bullet-point padding — that happens to score well. Removing the KL means removing the only pressure that keeps generations coherent.
3. It prevents mode collapse. A pure reward maximiser collapses toward a point mass because generation is sequential and errors compound. The KL keeps a floor under the policy’s support, so exploration does not die.

Modern plumbing note [as of 2026-10]. veRL makes the penalty a configuration choice: algorithm.use_kl_in_reward (documented default False), kl_penalty one of kl, abs, mse, low_var_kl, full, and a controller (kl_ctrl.type: fixed | adaptive, kl_coef documented as 0.005 in the example config but 0.001 in the current shipped config — the default has changed over time, so check your checkout — and target_kl: 0.1 for the adaptive variant). use_kl_in_reward defaulting to False is a real 2025–2026 change in convention: modern recipes either drop the KL for verifiable-reward domains (DeepSeek-V3.2 says verbatim that for “certain domains, such as mathematics, applying a relatively weak KL penalty or even omitting it entirely can yield improved performance”) or place it in the objective as GRPO does. Do not read the default as “the KL does not matter”; read it as “the frontier moved the leash”.

2.5 The full loss: value term and entropy bonus

The paper’s Equation 9 combines three terms:

$$L^{\text{CLIP+VF+S}}(\theta )={\hat{\mathbb{E}}}_t[L_t^{\text{CLIP}}(\theta )-c_1{L}_t^{\text{VF}}(\theta )+c_{2}S[{\pi }_{\theta }](s_t)]$$

$$L_t^{\text{VF}}(\theta )=(V_{\theta }(s_t)-V_t^{\text{targ}}{)}^2$$

• $V_{\theta }(s_t)$ — the critic’s prediction; $V_t^{\text{targ}}$ — the regression target (a GAE-based return, or the empirical return). The critic is trained by plain squared error; it is a separate head on the same backbone.
• $c_1$ — the value-loss coefficient. It is deliberately small: a badly-fitting critic must not be allowed to dominate the policy gradient. The N+ reproduction verifies $c_1=0.1$; the PPO paper defines $c_1$ without a single numeric default in its main text, so do not quote a “PPO default” for it. [UNVERIFIED: a canonical numeric default for $c_1$ in Schulman et al. 2017’s main text — the verified RLHF-era value is $0.1$.]
• $S[{\pi }_{\theta }](s_t)=-{\sum }_a{\pi }_{\theta }(a|s_t)\log {\pi }_{\theta }(a|s_t)$ — the entropy bonus, keeping the policy stochastic so exploration does not die. In LLM RL this is usually omitted or implicitly handled by the sampling temperature, a genuine difference from the control literature.
• $c_2$ — the entropy coefficient.
• $V_t^{\text{targ}}$ is commonly itself clipped (ε̂, the value clip range) so the critic’s target cannot move too far in one update. The N+ reproduction verifies value clipping on with $\widehat{ϵ}=0.2$. Small $c_1$ plus value clipping is the pair of choices that keeps a noisy critic from wrecking the run.

2.6 The loop, end to end

flowchart TB
    P["prompts (batch of 512)"] --> G["rollout from π_old<br/>vLLM / SGLang / HF generate"]
    G --> LP["store log π_old(a_t|s_t)"]
    G --> RM["reward model scores the completion"]
    RM --> KL["fold per-token KL into reward:<br/>r_t = r_RM·1[t=T] − β·(log π_θ − log π_ref)"]
    V["critic V_ψ forward pass"] --> GAE["GAE backward scan:<br/>δ_t = r_t + γV(s_{t+1}) − V(s_t)<br/>Â_t = δ_t + γλ·Â_{t+1}"]
    LP --> GAE
    KL --> GAE
    GAE --> UPD["K epochs: L = min(r_t·Â_t, clip(r_t,1−ε,1+ε)·Â_t) − c₁·(V−V_targ)²"]
    UPD --> SYNC["update actor θ and critic ψ"]
    SYNC --> G

The inner loop is the part that matters: after the rollout, the same batch is replayed for $K$ gradient epochs ($K=4$ in the verified N+ configuration) while ${\pi }_{{\theta }_{\text{old}}}$ stays frozen. Every epoch after the first is off-policy by construction, and the ratio plus the clip is what makes those epochs legitimate.

2.7 The four models

Plus the rollout engine that generates the completions (which in the naive case is the actor running autoregressively, and in modern frameworks is vLLM or SGLang sharing the GPUs). InstructGPT’s Figure 2 is this picture; OpenRLHF’s README states the placement consequence plainly: “It separates the Actor, Reward, Reference, and Critic models across different GPUs”. The next chapter, GRPO, exists almost entirely because of the critic row.

3. Implementation

3.1 TRL does not have PPO any more [as of 2026-10]

If you go looking for the canonical tutorial PPOTrainer, it is gone. On main, trl/trainer/__init__.py exports exactly SFTTrainer, DPOTrainer, KTOTrainer, GRPOTrainer, RLOOTrainer, RewardTrainer and DistillationTrainer (with matching *Config dataclasses). There is no ppo_trainer.py in the package. The deprecation was tracked in huggingface/trl issue #2016 (“Deprecate PPOTrainer”, opened and closed 2024-09-04), and as of TRL v1.15.0 (released 2026-10-08) there is no PPO trainer in the release. [UNVERIFIED: the exact release in which PPOTrainer was deleted — its absence from main/v1.15.0 and the deprecation issue are confirmed; the removal commit was not bisected.]

Practical reading. Do not present TRL as the PPO reference implementation. For PPO in 2026 you want OpenRLHF, veRL, DeepSpeed-Chat or SkyRL (see RL Frameworks).

3.2 PPO in a framework that actually implements it

OpenRLHF keeps PPO as its default advantage estimator. Its algorithm switch is --algo.advantage.estimator; leaving it at the default selects full-critic PPO. A skeleton on top of examples/scripts/ppo/ppo.sh, using only flags verified against the OpenRLHF README [as of 2026-10]:

# OpenRLHF (README recommends vLLM 0.29.0+ for rollout)
python3 -m openrlhf.cli.train_ppo_ray \
  --pretrain  meta-llama/Llama-3.1-8B-SFT \
  --reward_pretrain meta-llama/Llama-3.1-8B-RM \
  --ref_pretrain meta-llama/Llama-3.1-8B-SFT \
  --critic_pretrain meta-llama/Llama-3.1-8B-RM \
  --algo.advantage.estimator gae \
  --train.colocate_all \
  --vllm.num_engines 2 \
  --prompt_max_len 1024 --generate_max_len 4096 \
  --rollout_batch_size 512 \
  --actor_learning_rate 3e-6 --critic_learning_rate 9e-6 \
  --init_kl_coef 0.05 \
  --gamma 1.0 --lam 0.95 \
  --clip_eps 0.2 --value_clip_eps 0.2 --value_loss_coef 0.1 \
  --num_episodes 1 --max_epochs 4

Four things to notice, all of them PPO-specific:

• --critic_pretrain. The critic is a real file you have to supply, and the standard trick is to initialise it from the reward model — that is what InstructGPT did (“The value function is initialized from the RM”), because the RM already predicts a scalar per sequence and the value head is a related quantity. A randomly initialised critic is a large source of early instability.
• --init_kl_coef is the $\beta$ of Section 2.4. 0.05 is the N+ value; InstructGPT’s 0.02 is equally correct in its own codebase.
• --gamma 1.0 --lam 0.95 are the verified N+ settings. --lam 1.0 also appears in veRL’s documented example config; both are defensible, but they are different estimators and you should know which you are running.
• --train.colocate_all is the hybrid-engine placement: actor, critic, reward, reference and the vLLM engines time-share GPUs rather than each holding a full copy. Without it, a 7B PPO run needs a lot more hardware.

Flag names move between releases; treat this as a skeleton and check examples/scripts/ppo/ in the checkout you are actually using.

The placement knob, and one correctness knob. --train.colocate_all is the hybrid-engine idea: actor, critic, reward, reference and the vLLM engines time-share GPUs instead of each holding a full copy; veRL exposes the same idea as its hybrid engine, DeepSpeed-Chat as the DeepSpeed Hybrid Engine. The correctness knob is importance-sampling correction (rollout_is_threshold: 2.0 in veRL). The log-probabilities produced by the fast inference engine do not exactly match the training engine’s, and DeepSeek-V3.2 states the consequence: “Such training-inference inconsistency further exacerbates the degree of off-policyness.” Across two engines this is not optional bookkeeping.

3.3 The two functions you should be able to write from memory

Framework-free reference implementation, runnable as pseudocode — the point is the shapes, not the plumbing:

import torch

def gae(rewards, values, gamma=1.0, lam=0.95):
    """rewards, values: (T,) per-token reward and V(s_t).
    Returns advantages Â_t (T,) and value targets (T,).
    """
    T = len(rewards)
    adv = torch.zeros(T, dtype=values.dtype)
    last = 0.0
    for t in reversed(range(T)):                 # backward scan, O(T)
        next_v = values[t + 1] if t + 1 < T else 0.0
        delta = rewards[t] + gamma * next_v - values[t]      # TD residual
        last = delta + gamma * lam * last                    # Â_t = δ_t + γλ·Â_{t+1}
        adv[t] = last
    return adv, (adv + values).detach()          # targets = Â_t + V(s_t), no grad

def ppo_loss(logp, logp_old, advantages, values, values_old, returns,
             clip_eps=0.2, value_clip_eps=0.2, c1=0.1):
    ratio = torch.exp(logp - logp_old)                    # r_t(θ) = π_θ / π_old
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * advantages
    policy_loss = -torch.min(unclipped, clipped).mean()   # note the MIN
    # Clip the critic's prediction around V_old, the value stored with the rollout
    # -- the same fixed anchor logp_old is for the ratio. Clipping `values` around
    # itself (values - values.detach()) is identically zero and does nothing.
    values_clipped = values_old + (values - values_old).clamp(
        -value_clip_eps, value_clip_eps)
    value_loss = torch.max((values - returns) ** 2,
                           (values_clipped - returns) ** 2).mean()
    return policy_loss + c1 * value_loss

Three details that interviewers check. First, torch.min — not max, not an average. Second, the ratio is computed in log space and exponentiated; doing it as pi / pi_old directly will overflow in fp16 once the log-ratio drifts past a few, whereas exp(logp - logp_old) stays finite until much further out (and you should be clipping anyway). Third, the value clip is anchored on values_old, the critic prediction stored with the rollout, exactly as the ratio is anchored on logp_old; clipping the current prediction against itself is a no-op. Also note that values here is from the current forward pass while returns came from the rollout’s GAE and is detached, so the critic loss has exactly one gradient path into the value prediction — mixing up “value of the state under the current policy” and “target”, or leaving the target attached, is a classic silent bug.

4. Numbers and trade-offs

4.1 The memory bill for four models

Derive it rather than quoting it, because the number depends on your configuration. Take a 7B actor in bf16 with Adam (fp32 moments) and a gradient buffer. Per parameter, a trainable copy costs:

$$2\text{(bf16 weights)}+2\text{(bf16 grads)}+4+4\text{(fp32 }m,v)=12\text{bytes}$$

An inference-only copy costs $2$ bytes (bf16 weights) and no optimizer state. For 7B parameters:

[as of 2026-10] This is a stated configuration, not a universal constant: ZeRO-3 shards it across devices, CPU offload moves the optimizer state off the GPU, and the reference’s per-token log-probabilities can be precomputed once and cached, which removes the reference model from the live set entirely. The honest version of the claim is structural: PPO needs an actor-sized trainable critic with its own optimizer state, and that is the single largest avoidable allocation. [UNVERIFIED: the byte-level budget above is derived in-line from the stated precision choices, not cited, per the research note that no source gives this arithmetic.]

This section counts $12$ bytes per trainable parameter, without a standalone fp32 master copy. RL Frameworks §2.1 uses the $16$-byte convention that includes the master copy and therefore derives a 252 GB PPO total for the same configuration; its reconciliation note covers the difference, and the two tables should not be mixed.

4.2 Where the wall clock goes

OpenRLHF’s README states “RLHF training spends 80% of the time on sample generation” (project self-report, not an independent measurement). The qualitative claim is forced by the algorithm: PPO is on-policy, so every gradient cycle needs fresh completions, and generating 4,000 tokens costs vastly more than one forward pass over them. $K=4$ epochs on that batch is the only amortisation PPO gets. This is the axis on which GRPO (which samples $G$ completions per prompt) and asynchronous systems are trading.

4.3 The verified hyperparameter table

From the N+ reproduction (Huang et al. 2024, arXiv:2403.17031, TL;DR summarization). Use these as the realistic range instead of inventing numbers:

Compare InstructGPT: $\beta =0.02$, batch 512 / minibatch 64, 256k episodes. A 10x difference in $\beta$ between two correct implementations is normal — it is not a tuned constant of nature.

5. Pitfalls

Reward hacking / KL collapse. Symptom: proxy reward climbs smoothly while the gold or held-out score peaks and then falls; generations get longer, more bullet-pointed, more obsequious; refusal rate collapses. Cause: the policy has walked off the RM’s training distribution. Fix: larger $\beta$, early stopping on a frozen gold set, RM ensembling, or re-labelling — not a bigger $ϵ$.

Ratio explosion from stale data. Symptom: the ratio histogram grows a fat right tail across the $K$ epochs; the clip fraction rises from a few percent to most of the batch. Cause: too many epochs on one batch, or the training engine’s log-probabilities disagreeing with the inference engine’s. Fix: fewer epochs, lower lr, and IS truncation at 2.0 (rollout_correction in veRL).

Critic dominates the policy update. Symptom: value loss falls nicely while the policy does not move. Cause: $c_1$ too large, or the critic’s target is unclipped so it chases. Fix: $c_1=0.1$ or lower, value clipping on, critic initialised from the RM.

Truncated sampling breaks importance sampling. Symptom: training is unstable in a way that no learning-rate change fixes, specifically when top-p or top-k sampling is on. Cause: as DeepSeek-V3.2 puts it, top-p/top-k “introduces a mismatch between the action spaces of ${\pi }_{\text{old}}$ and ${\pi }_{\theta }$, which violates the principles of importance sampling” — a token with zero sampling probability gets a nonzero training probability, so the ratio is unbounded. Fix: keep the sampling mask from the rollout and apply it during training (DeepSeek-V3.2’s “Keep Sampling Mask”).

Temperature 1.0+. Symptom: the reward is flat at the floor and the value loss never converges; half the samples are incoherent. Cause: the critic is being asked to predict the return of noise. Fix: N+ verified 0.7; higher temperatures (0.9–1.0) are normal for RLVR reasoning, but then the baseline must come from the group, not a critic.

Unescaped / mismatched log-probs. Symptom: NaN loss on the first backward pass; exp(logp - logp_old) overflow. Cause: fp16 log-softmax over a long sequence, or a log-prob recomputed with a different tokenizer/padding scheme than the rollout used. Fix: apply the length mask consistently, compute the ratio in fp32, and assert that at epoch 0 the mean ratio is 1.0.

Discounting the terminal reward. Symptom: long correct answers are penalised relative to short ones for no stated reason; the model prefers terse outputs. Cause: $\gamma <1$ on a bounded episode. Fix: $\gamma =1.0$.

6. Using it in a project

Reach for PPO when the reward is dense or shaped, and when you need a critic. The value function earns its memory exactly when there is structure to learn — turn-level rewards in a multi-turn agent, a process reward model, a reward with partial credit, or a shaping term you control. VAPO (arXiv:2504.05118) is the strongest published counter-current to the critic-free trend: it adds the value model back with fixes for value-model bias, heterogeneous sequence lengths and sparse rewards, and reports 60.4 on AIME 2024 with a Qwen-32B base (self-reported). If someone tells you value-based RL is dead, that is the citation that says otherwise.

Do not reach for PPO when:

• Your preference data is thin or your budget is one SFT run. Use SFT. No amount of PPO repairs a reward signal built from a few hundred comparisons.
• The reward is a binary verifier. Then the value function is being trained to predict a coin flip, and a group mean does that job for free. This is the GRPO case, and it is where the field went.
• You cannot pay for on-policy generation. PPO is on-policy: no new completions, no gradient. If sampling is the binding constraint, DPO (DPO) turns the same preference data into a classification loss with no environment at all.
• You are compute-constrained to a single 7B-class GPU. Four 7B models is the arithmetic in Section 4.1; a colocated framework can make it fit, but a smaller or critic-free method will simply run.
• You need the rollout to be reproducible. Critic-free outcome-reward methods have far fewer stochastic parts to debug.

The 2026 framing, stated plainly. PPO-with-critic is legacy as the default, not dead. [as of 2026-10] The frontier default for reasoning post-training is critic-free group-relative RL against verifiable rewards (DeepSeek-V3.2’s §3.1 is titled “Scaling GRPO”; Qwen3 §4.2 names GRPO; DAPO/GSPO are GRPO-adjacent; VAPO is the value-based counter-current). PPO is still the right tool when the reward is dense and the value estimate is worth fitting, and every major framework ships a critic path. What changed is which method is the default — and, per Section 3.1, that TRL no longer ships one at all.

7. How you would prove it worked

PPO has no product metric of its own; you prove that a design choice matters.

The KL ablation. Two runs, $\beta =0$ versus $\beta =0.05$ (example), all else fixed. Plot proxy reward, gold reward and $d=\sqrt{D_{KL}({\pi }_{\theta }\Vert {\pi }_{\text{init}})}$ against step. Expect the Gao et al. shape: the $\beta =0$ run’s proxy rises faster and its gold score turns over earlier. Report the step at which gold peaks for each. The claim is “the KL penalty moved the gold-score peak later”, not “the KL penalty made the model better” — Gao et al. is explicit that it does not raise the frontier.

The ratio histogram as a health check. Log the fraction of tokens with $|r_t-1|>ϵ$ per epoch and the ratio’s 99th percentile. (example) A healthy run clips a few percent in epoch 1 and rises with $K$; a run that starts above 20% has too many epochs, too high an lr, or a train/inference log-prob mismatch. This cheap plot separates “my policy is learning” from “my estimator is broken”.

The critic ablation. Compare $\lambda =0.95$ against $\lambda =0$ on a reward with real intermediate structure (a shaped multi-turn reward, not a terminal verifier). (example) With a terminal-only reward, $\lambda =0$ is usually worse because the freshly-initialised value head is wrong at short horizons. The experiment tells you whether the critic is earning its 84 GB.

Reporting template (example — replace with your own numbers). “With $\beta =0$ the held-out preference win-rate peaked at step 1,200 and fell 4 points by step 4,000; with $\beta =0.05$ it peaked later and held, and the clipped-token fraction stayed under 6%. We kept $\beta =0.05$ and dropped the run that looked best on proxy reward.”

8. Interview drill

“Why clip the ratio at all?” Strong answer: it bounds how far one update can move the policy from the data-generating policy. Without it, multi-epoch updates on one batch are biased and the policy can leave the reward model’s region of validity in a single step; with it, moving outside the region earns no gradient, so the objective is a pessimistic lower bound on the improvement — a cheap first-order stand-in for TRPO’s KL constraint. Common wrong answer: “clipping prevents the ratio from overflowing numerically.” That is a side effect; an unbounded-ratio example can never overflow. The reason is the trust region, not float range.

“Why min and not max?” Strong answer: min makes the objective pessimistic — the gradient switches off exactly when the policy is already outside the trust region and still pushing further in the advantageous direction (good action, $r>1+ϵ$; bad action, $r<1-ϵ$). max would reward leaving the region, the opposite of a trust region. Common wrong answer: “min keeps the loss from getting too large.” It is a statement about incentives, not magnitudes. A good follow-up is “draw the objective as a function of $r_t$ for $\hat{A}>0$” — the correct picture is a line of slope $\hat{A}$ that flatlines at $r=1+ϵ$.

“Where exactly does the KL term enter?” Strong answer: most LLM implementations fold it into the per-token reward as $-\beta (\log {\pi }_{\theta }-\log {\pi }_{\text{ref}})$ before GAE, so it flows through the advantage; GRPO instead writes a separate $-\beta D_{KL}$ term inside the objective. Either way it is a per-token penalty against a frozen reference, and the reference is mandatory. Common wrong answer: “the KL is the loss function” or “the KL is what PPO clips.” The clip operates on the advantage-weighted ratio; the KL is a separate distance penalty.

“Why does PPO need a critic, and what does it cost?” Strong answer: the critic supplies a per-token baseline for ${\hat{A}}_t$; without a baseline the estimator’s variance is the variance of the return. It costs a second trainable model at actor scale, with its own optimizer state, gradients and forward/backward passes — deriving the 12-bytes-per-param versus 2 accounting shows you have budgeted a run. It is also the hardest model to fit on long chain-of-thought, which is precisely why GRPO deletes it. Follow-up: “what initialises the critic?” → the reward model, per InstructGPT.

“$\lambda =0$ versus $\lambda =1$?” Strong answer: $\lambda =0$ is the one-step TD residual — low variance, biased whenever $V$ is wrong. $\lambda =1$ is the empirical return minus the value baseline — unbiased given the baseline, high variance because it sums the remaining rewards. $0.95$ is the usual compromise because the value head is unreliable early. Common wrong answer: “$\lambda$ is the discount factor.” That is $\gamma$; in GAE they always appear as the product $\gamma \lambda$, which is exactly why $\lambda$ does not change the horizon.

“Why is $\gamma =1$ here when RL always uses $\gamma <1$?” Strong answer: the episode is one bounded response and the reward is at EOS; discounting would make the objective depend on response length and would shrink the only reward you have. Control papers discount because episodes are unbounded. Common wrong answer: “because the reward is at the end, the discount does not matter.” With a per-token KL penalty in the reward it does, and the standard choice is still $\gamma =1$.

“Is PPO still what people use in 2026?” Strong answer: legacy as the default, not dead. [as of 2026-10] DeepSeek-V3.2’s §3.1 is titled “Scaling GRPO”, Qwen3 §4.2 names GRPO, DAPO/GSPO are GRPO-adjacent, and TRL has removed its PPO trainer entirely (issue #2016). PPO remains right where rewards are dense and a value estimate is worth fitting — VAPO’s 60.4 on AIME is the published counter-current — and every major framework still ships a critic path. Common wrong answer: “PPO was superseded, nobody uses it.” That ignores VAPO and the dense-reward cases, and it ignores that GRPO is PPO’s clipped surrogate with the advantage replaced.

Next: GRPO — PPO minus the critic: the same clipped surrogate, with the learned baseline swapped for the mean reward of a group of samples.