GRPO

TL;DR — Group Relative Policy Optimization is PPO with the critic deleted. Instead of training a value network to answer “was this token better than average?”, GRPO samples a group of $G$ completions for the same prompt, scores each one, and uses the group’s own reward statistics as the baseline: ${\hat{A}}_{i,t}=(r_i-\mathrm{mean}(r))/\mathrm{std}(r)$, a single scalar shared by every token of completion $i$. That removes the value model, its optimizer state and all of GAE — PPO’s four resident models become two (actor plus frozen reference) — at the price of multiplying rollout tokens by $G$. It is the algorithm that made RL with verifiable rewards practical: [as of 2026-10] DeepSeek-V3.2’s flagship post-training section is titled “Scaling GRPO”, Qwen3 §4.2 names GRPO, and the DAPO/GSPO/Dr. GRPO family are all GRPO edits. Its known biases are the per-sequence length normalisation and the local standard-deviation normalisation; Dr. GRPO removes both.

1. Why this exists

PPO’s critic is the most expensive and least reliable part of the algorithm. Two sentences from the PPO chapter are the whole motivation:

1. The critic is usually actor-sized — a second trainable 7B model with its own gradients and Adam state.
2. The advantage is computed by bootstrapping through that critic’s estimate, so the critic’s error is the bias of ${\hat{A}}_t$.

Now make that concrete with the workload GRPO was built for: a maths problem, a verifiable answer, and a reward that is $1$ if the final answer matches and $0$ otherwise. What should $V(s_t)$ predict? It should predict the probability that the model, from this partial solution, will eventually reach a correct answer. That is a genuinely hard quantity to regress, and early in training the freshly initialised value head is close to noise. With GAE at $\lambda =0.95$, that noise is mixed into every token’s advantage. Meanwhile the reward the policy cares about — did the answer come out right — is available exactly, for free, from a checker. PPO spends an actor-sized model to estimate it, and estimates it badly.

There is a second, subtler failure that appears at long horizons. Reasoning traces are thousands to tens of thousands of tokens. A value head must be accurate at every prefix to give a useful bootstrap, and the error compounds along the trace. At the extreme, the critic is being asked to predict a return from a state deep inside a chain of thought that the model itself cannot reliably evaluate.

GRPO’s observation is that you do not need a learned baseline if you can afford to sample several completions of the same prompt. Draw $G$ completions, score them all, and the mean reward of that group is an unbiased estimate of the policy’s average performance on this exact prompt — which is precisely what the baseline is supposed to be. No critic, no bootstrapping, no value-loss coefficient, no GAE. The credit assignment becomes deliberately coarse: a whole completion is reinforced or suppressed based on how it compared to its siblings.

Why this only became the default when it did. Group sampling is only cheap when the reward is cheap. A reward model can score $G$ completions too, but the economics change completely when the reward is a string match or a unit test: the marginal cost of scoring the group is essentially zero, and the only real cost is generation. That is why GRPO’s rise is inseparable from RLVR — Reinforcement Learning with Verifiable Rewards, the term introduced by Tulu 3 (Lambert et al. 2024, arXiv:2411.15124).

GRPO was introduced in DeepSeekMath (Shao et al. 2024, arXiv:2402.03300, §4.1.2) and became famous through DeepSeek-R1 (DeepSeek-AI 2025, arXiv:2501.12948; journal version Nature 645:633–638).

2. How it works

2.1 The group-relative advantage, symbol by symbol

Everything distinctive about GRPO is in this one line. For a prompt $q$, sample $G$ completions $\{o_1,\dots ,o_G\}$ from the current policy, score each with the reward function to get $r_i=r(q,o_i)$, then define:

$${\hat{A}}_{i,t}={\tilde{r}}_i=\frac{r_i-\mathrm{mean}(r)}{\mathrm{std}(r)},r=(r_1,\dots ,r_G)$$

• $q$ — the prompt (DeepSeekMath writes the question distribution as $P(Q)$ and samples $q\sim P(Q)$). One prompt per group; the group is the unit of comparison.
• $G$ — the group size: how many completions are sampled per prompt. It is the parameter TRL exposes as num_generations (self.num_generations = args.num_generations # = G in the GRPO paper). $G$ is the dial that trades rollout compute for baseline quality: with $G=1$ the baseline is the reward itself and the advantage is identically zero, so GRPO degenerates into nothing; the variance of the baseline estimate falls like $1/G$.
• $o_i$ — the $i$-th sampled completion; $\{o_1,\dots ,o_G\}$ are i.i.d. draws from ${\pi }_{{\theta }_{\text{old}}}(\cdot \mid q)$. This is the only place the group enters; there is no cross-completion attention or comparison model.
• $r_i=r(q,o_i)$ — the scalar reward of completion $i$. A single number per completion, from a verifier, an RM, or a mixture. GRPO needs no reward at any intermediate token.
• $\mathrm{mean}(r)$ — the baseline. It estimates $V^{\pi }(q)$, the policy’s average reward for this prompt. Subtracting a baseline is legal because ${\mathbb{E}}_{a\sim \pi }[A^{\pi }(s,a)]=0$: the expected policy gradient is unchanged while the variance drops. This term is the replacement for the critic. Remove it and every completion of a group is pushed up (or down) together, which is a pure variance term that teaches the model nothing about which sibling was better.
• $\mathrm{std}(r)$ — the scale normaliser, the standard deviation of the group’s rewards. It makes the advantage dimensionless, so a prompt whose rewards are all within $[0.9,1.0]$ produces the same shape of update as one whose rewards range over $[0,1]$; without it you would need a per-task learning-rate schedule. It is also the term Dr. GRPO identifies as biased (Section 2.6).
• ${\tilde{r}}_i$ — the group-relative advantage. $+1$ means “roughly one standard deviation better than this prompt’s average”; the scale is in units of the group’s own spread, which is exactly the quantity with which you want to weight a policy-gradient step.
• ${\hat{A}}_{i,t}$ — the advantage indexed by token $t$, but note the decisive detail: the right-hand side does not depend on $t$ at all. Every token of completion $i$ receives the same advantage. There is no per-token credit assignment beyond “this whole response was better than its siblings”. That is a deliberate trade of granularity for stability — and it is why the reward must be meaningful for a complete response, which a verifier’s answer-match is and a process reward model is not.

Two degenerate groups, and one division by zero. If all $G$ rewards are equal, $\mathrm{std}(r)=0$ and the advantage is 0/0. Mathematically the group carries no signal — the policy did equally well on every sibling, so there is nothing to reinforce — and implementations either skip it or add a small epsilon. But the symptom is worth knowing: at the start of a hard task every group is all-wrong (all rewards $0$), and at the end every group is all-right (all rewards $1$). Those prompts produce zero gradient and the rollout compute spent on them is wasted. DAPO’s dynamic sampling exists precisely to drop all-0 and all-1 groups (arXiv:2503.14476).

2.2 The objective

DeepSeekMath’s Equation 24, with the notation above:

$$J_{\text{GRPO}}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _{i=1}^G\frac{1}{|o_i|}\sum _{t=1}^{|o_i|}\min \right(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{\text{old}}}(o_{i,t}\mid q,o_{i,<t})}{\hat{A}}_{i,t},\mathrm{clip}(\cdot ,1-ϵ,1+ϵ){\hat{A}}_{i,t})-\beta D_{KL}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}\left]\right]$$

• $o_{i,t}$ — the $t$-th token of completion $i$; $o_{i,<t}$ — its prefix. The state is prompt plus prefix, exactly as in PPO.
• ${\pi }_{\theta }/{\pi }_{{\theta }_{\text{old}}}$ — the same probability ratio as PPO, with ${\pi }_{{\theta }_{\text{old}}}$ the snapshot that generated the group. GRPO is a clipped ratio method: remove the ratio and clip and you are left with a plain REINFORCE-style estimator, which is what distinguishes GRPO from RLOO and REINFORCE++ (Section 2.6). The clip range $ϵ$ means the same thing it does in PPO: the ratio may move at most a factor $1+ϵ$ up or $1-ϵ$ down before the gradient for that token switches off, and the min makes the objective pessimistic (it never rewards leaving the trust region).
• $1/|o_i|$ — the per-sequence normalisation. Each token’s loss is divided by the length of its own response, so that ${\sum }_{t}1/|o_i|=1$: each completion contributes a fixed budget regardless of how long it is. This is the sequence-level aggregation choice. It is also the length bias Dr. GRPO attacks (Section 2.5).
• $1/G$ — the per-group normalisation, averaging the $G$ completion contributions so a prompt contributes one unit regardless of group size.
• $\beta D_{KL}[{\pi }_{\theta }\Vert {\pi }_{\text{ref}}]$ — the KL penalty, written inside the objective rather than folded into the reward as PPO does. Same effect, different plumbing: it is now visible to the same clamp/min machinery. ${\pi }_{\text{ref}}$ is the frozen reference (usually the SFT checkpoint) and $\beta$ is the same kind of coefficient as PPO’s, with the same meaning: how many reward units one nat of divergence costs. The reference model is not removed by GRPO. DeepSeek-V3.2 §3.1 does note that “for certain domains, such as mathematics, applying a relatively weak KL penalty or even omitting it entirely can yield improved performance”, and veRL’s documented default is use_kl_in_reward: False — but that is a tuning decision, not a structural one.
• $\mathbb{E}[\cdot ]$ — over prompts $q$ and groups sampled from ${\pi }_{{\theta }_{\text{old}}}$. The empirical version averages over all tokens of all completions in the batch.

The whole objective is PPO’s clipped surrogate with two differences: $\hat{A}$ is group-relative instead of GAE, and the KL is a term in the loss. Everything else — ratio, clip, min, epochs over one batch — is unchanged. GRPO is “PPO minus the critic”, not “PPO minus the RL”.

2.3 What is actually computed: the gradient form

DeepSeekMath also gives the gradient (its Equation 25), which is the most honest statement of what the code does:

$${\nabla }_{\theta }{J}_{\text{GRPO}}(\theta )=\mathbb{E}\left[\frac{1}{G}\sum _i\frac{1}{|o_i|}\sum _t\right({\hat{A}}_{i,t}+\beta ({\pi }_{\text{ref}}(o_{i,t}\mid o_{i,<t})/{\pi }_{\theta }(o_{i,t}\mid o_{i,<t})-1)){\nabla }_{\theta }\log {\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})]$$

This is REINFORCE with the group-relative advantage as the weight, plus a per-token KL correction. Two things to read off it:

• The bracket is the effective per-token weight. DeepSeekMath’s appendix writes the same combination as $GC={\hat{A}}_{i,t}+\beta ({\pi }_{\text{ref}}/{\pi }_{\theta }-1)$ for comparison with PPO.
• The term ${\pi }_{\text{ref}}/{\pi }_{\theta }-1$ is the $k_3$ KL estimator’s gradient (Schulman 2020, joschu.net/blog/kl-approx.html), which is non-negative and lower-variance than the naive log-ratio. DeepSeek-V3.2 later corrects it for off-policyness by multiplying by the importance ratio ${\pi }_{\theta }/{\pi }_{{\theta }_{\text{old}}}$, because the uncorrected $k_3$ gradient “assigns disproportionately large, unbounded weights” when ${\pi }_{\theta }\ll {\pi }_{\text{ref}}$ (arXiv:2512.02556, §3.1).

2.4 Exactly what is removed versus PPO

Not removed: the reference model, the reward function (learned or programmatic), the ratio, the clip, the min, and the sampling loop.

2.5 The memory consequence

This is the concrete win, derived for the same configuration as the PPO chapter: a 7B actor in bf16, Adam with fp32 moments, a gradient buffer. A trainable copy costs $2+2+4+4=12$ bytes per parameter; an inference-only copy costs $2$.

[as of 2026-10] Stated configuration, not a universal constant: ZeRO sharding and CPU offload change the numbers, and the reference’s log-probabilities can be cached so the reference is not necessarily live. The structural claim is the one that matters: GRPO removes an actor-sized trainable model and its optimizer state, and adds generation. Against that saving you pay $G\times$ the rollout tokens — and since rollout is the dominant wall-clock term (OpenRLHF’s README reports “RLHF training spends 80% of the time on sample generation”, a project self-report), the trade is memory and critic-fitting difficulty down, generation compute up. GRPO wins exactly where memory is the binding constraint and the reward is cheap to evaluate — which is RLVR maths and code, and is why it took over there first. [UNVERIFIED: the byte-level arithmetic above is derived in-line from the stated precision choices; no source gives this budget directly.]

This table uses the PPO chapter’s $12$-bytes-per-trainable-parameter convention. RL Frameworks §2.1 adds the fp32 master copy, giving $16$ bytes, and for the same configuration quotes 252 GB for PPO and 140 GB for GRPO; the reconciliation note there explains the gap, and the two tables should not be mixed.

2.6 The biases, and the variants that fix them

Dr. GRPO — the name is introduced inside Liu et al. 2025, Understanding R1-Zero-Like Training: A Critical Perspective (arXiv:2503.20783). The paper identifies two structural biases in GRPO’s loss:

1. Length-normalisation bias. Dividing each completion’s loss by $1/|o_i|$ means a long, wrong response has its per-token penalty diluted. The objective is quietly indifferent to length, so the policy drifts longer, and the effect is worst for incorrect outputs, which is the opposite of what you want. The fix is to drop that per-completion denominator and divide by a constant (Dr. GRPO uses the maximum completion length) so every token carries the same weight. A std-removal flag does not touch this bias — it is a separate change.
2. Standard-deviation bias. $\mathrm{std}(r)$ is a local, per-prompt statistic. A prompt whose group happens to have low reward variance — including one where every answer is wrong — gets its advantage inflated by the small denominator. REINFORCE++ (Hu et al. 2025, arXiv:2501.03262) makes the same argument formally: prompt-level local normalisation is “a theoretically biased estimator”, and its fix is Global Advantage Normalization, which is “effectively unbiased” with bias vanishing as the batch grows. OpenRLHF exposes the local-std half as --algo.advantage.estimator dr_grpo, described in its README as “Removes local /std norm”; that flag does not by itself remove the length normalisation of item 1.

Dr. GRPO is “an unbiased optimization method that improves token efficiency while maintaining reasoning performance”, reporting 43.3% on AIME 2024 from a 7B base (self-reported). The same paper is also a caution about attribution: it finds that DeepSeek-V3-Base “already exhibit[s] ‘Aha moment’” and that Qwen2.5 base models reason well “even without prompt templates, suggesting potential pretraining biases” — some of what was credited to GRPO was already in the base model or the prompt format.

The other name-bearing variants, all GRPO-adjacent and all built on veRL:

Note the shared failure these all attack: entropy collapse during long RL runs (arXiv:2505.22617).

2.7 The loop

flowchart TB
    Q["prompt batch"] --> S["sample G completions per prompt<br/>from π_old (G = num_generations)"]
    S --> R["score each completion once:<br/>verifier gives r_i in 0 or 1, or RM gives r_i"]
    R --> N["group normalise:<br/>Â_i = (r_i − mean(r)) / std(r)<br/>broadcast the scalar to every token"]
    N --> L["clipped surrogate per token:<br/>min(ratio·Â_i, clip(ratio,1−ε,1+ε)·Â_i)<br/>− β·D_KL[π_θ ‖ π_ref]"]
    L --> A["1/|o_i| then 1/G aggregation"]
    A --> U["gradient step on θ"]
    U --> S

The shape to notice: one reward number per completion, broadcast to a whole response. There is no path from an individual token to an individual reward, which is both the simplification and the limitation of the method.

3. Implementation

3.1 TRL GRPOTrainer

TRL is the fastest way to run GRPO on one node [as of 2026-10], TRL v1.15.0. The trainer takes a reward function (or a list of them, each a Python function, an async function, an nn.Module, or a path to a sequence-classification reward model) rather than reward-model logits:

# trl >= 1.15.0, transformers, torch; single-GPU or DDP example
from datasets import load_dataset
from trl import GRPOConfig, GRPOTrainer

import re

def correctness_reward(completions, answer, **kwargs):
    """Verifiable reward: 1.0 if the boxed final answer matches, else 0.0.
    One scalar per completion -- GRPO never looks inside the response."""
    rewards = []
    for completion, gold in zip(completions, answer):
        m = re.search(r"\\boxed\{(.*?)\}", completion[-300:])
        rewards.append(1.0 if (m and m.group(1).strip() == gold.strip()) else 0.0)
    return rewards

def format_reward(completions, **kwargs):
    """A small dense shaping term: encourages the answer to be well formed."""
    return [0.1 if "\\boxed{" in c else 0.0 for c in completions]

dataset = load_dataset("openai/gsm8k", "main", split="train")

config = GRPOConfig(
    output_dir="grpo-gsm8k",
    # ---- the group ----
    num_generations=8,          # = G in the GRPO paper
    generation_batch_size=64,   # generations decoded per rollout; must be divisible by G
    max_completion_length=1024, # truncation teaches nothing if it fires often
    temperature=0.9,            # G samples must differ, or std(r) = 0
    # ---- the surrogate ----
    loss_type="grpo",           # the surrogate in Section 2.2; other variants selectable
    scale_rewards=True,         # divide by std(r); False drops the local std term (part of Dr. GRPO)
    importance_sampling_level="token",  # 'sequence' approximates GSPO's ratio
    beta=0.04,                  # KL coefficient to the reference, if used
    num_iterations=1,           # >1 reuses a rollout for several updates (off-policy)
    # ---- optimisation ----
    learning_rate=1e-6,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    bf16=True,
)

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    reward_funcs=[correctness_reward, format_reward],
    args=config,
    train_dataset=dataset,
)
trainer.train()

The knobs that actually matter, with what moving each does:

• num_generations ($G$) — the baseline’s sample size. Larger $G$ means a lower-variance baseline and more chances of seeing a success on a hard prompt; it multiplies rollout cost linearly. It must divide generation_batch_size — a mismatch is a hard error, not a silent fallback.
• scale_rewards — when True, divide by std(r). Set it False to remove the local std normalisation — the standard-deviation half of the Dr. GRPO fix. Dr. GRPO also drops the $1/|o_i|$ length normalisation, which this flag does not touch, so False alone is a partial ablation, not the whole method.
• loss_type — which surrogate to use (grpo is the form in Section 2.2). This is where DAPO-style variants live.
• importance_sampling_level — token is GRPO’s per-token ratio; sequence lifts it to the sequence likelihood, which is GSPO’s central idea.
• beta — the KL coefficient. Zero disables the reference term; see the DeepSeek-V3.2 caveat that weak or zero KL can help in maths.
• num_iterations — gradient updates per rollout. Above 1 this is the off-policy regime, exactly as PPO’s $K$ epochs are.
• temperature — with temperature too low, the $G$ samples are nearly identical, $\mathrm{std}(r)\approx 0$, and the group produces no signal. This is the single most common cause of “GRPO does nothing”.

3.2 veRL

veRL’s GRPO is selected by the advantage estimator, not by a separate trainer [as of 2026-10]. Note the repository moved: volcengine/verl now redirects to github.com/verl-project/verl, which is the canonical URL for new links:

algorithm:
  adv_estimator: grpo       # gae = PPO with a critic; grpo = group-relative, no critic
  gamma: 1.0                # unused by grpo: the advantage is a group statistic
  use_kl_in_reward: False   # documented default; KL is handled in the loss
actor_rollout_ref:
  rollout:
    n: 8                    # group size (rollout multiplier); check the key spelling
                            # against your checkout -- the GRPO doc writes it
                            # inconsistently with the rest of veRL
reward_model:
  enable: False             # maths/GSM8K examples use rule-based rewards

veRL’s own GRPO doc describes the mechanism in the same three bullets: “No Value Function (Critic-less)”, “Group Sampling (Grouped Rollouts)”, “Relative Rewards: within each group, completions are scored (e.g. based on correctness), and rewards are normalized relative to the group.” [UNVERIFIED: the exact spelling of the group-size key — the veRL GRPO doc writes actor_rollout.ref.rollout.n while the rest of veRL uses the actor_rollout_ref.* prefix; check verl/trainer/config/*.yaml before copying a config.]

3.3 The two functions, from memory

import torch

def group_relative_advantage(rewards, eps=1e-4, scale=True):
    """rewards: (num_prompts, G) -> advantages of the same shape.
    Every completion in a group is compared only to its own siblings.
    """
    mean = rewards.mean(dim=1, keepdim=True)
    adv = rewards - mean                       # baseline subtraction
    if scale:
        adv = adv / (rewards.std(dim=1, keepdim=True) + eps)   # local std (Dr. GRPO removes this)
    return adv

def grpo_loss(logp, logp_old, adv_per_token, logp_ref=None, beta=0.0,
              clip_eps=0.2, lengths=None):
    """logp, logp_old, adv_per_token, logp_ref: (B, T) flattened over
    completions x tokens. adv_per_token is constant along T within a completion.
    lengths: (num_completions,) for the 1/|o_i| normalisation.
    """
    ratio = torch.exp(logp - logp_old)                     # π_θ / π_old, computed in log space
    unclipped = ratio * adv_per_token
    clipped = torch.clamp(ratio, 1 - clip_eps, 1 + clip_eps) * adv_per_token
    per_token = -torch.min(unclipped, clipped)
    if logp_ref is not None and beta:
        per_token = per_token + beta * (logp - logp_ref)   # D_KL term, non-negative form is better
    # 1/|o_i| inside the sum, then 1/G across the group:
    seq_loss = torch.stack([per_token[i, :n].sum() / n for i, n in enumerate(lengths)])
    return seq_loss.mean()

Two implementation facts worth remembering. The ratio must be computed as exp(logp - logp_old), never as a direct division — the same reason as in PPO. And the 1/|o_i| normalisation is a real algorithmic choice with a measured bias, so if you change it, log response length as a training curve; the length drift is the symptom.

4. Numbers and trade-offs

4.1 Memory versus rollout: the actual trade

The comparison that matters is per unit of rollout, not per algorithm. If PPO takes one sample per prompt and GRPO takes $G$, GRPO sees $G\times$ the completions for the same number of prompts. To compare fairly you either hold generated tokens constant (GRPO gets $1/G$ the prompts, but each group’s baseline is better) or hold prompts constant (GRPO generates $G\times$ more). Which is better is empirical and depends on $G$ and on how much a good baseline is worth. The reported benefit — lower memory, no critic to fit — is real; the “GRPO is cheaper” claim is not true in rollout FLOPs.

4.2 Group size

$G$ sets the variance of the baseline like $1/G$, but the returns flatten fast. (example) Going from $G=1$ to $G=8$ buys most of the available variance reduction; going from $8$ to $64$ costs 8x the generation for a small further gain. Small $G$ also raises the probability that a hard prompt yields an all-wrong group and therefore no gradient at all — which is the regime dynamic sampling is for. Values in the 4–64 range are common in published recipes, with 8 a frequent default; pick by measuring your zero-variance-group rate, not by copying a number. [UNVERIFIED: no single canonical $G$ — DeepSeekMath’s exact value was not extracted from the paper.]

4.3 Entropy and length are the curves to watch

GRPO’s two documented failure modes both show up as clean scalar curves: response length drifts upward (length bias) and policy entropy collapses (arXiv:2505.22617). Every variant in Section 2.6 is an attempt to move one of those two curves. If you log only reward, both failures look like success for a while.

5. Pitfalls

All-zero or all-one groups. Symptom: reward improves but the gradient norm is frequently zero; a large fraction of rollouts produce no update. Cause: $\mathrm{std}(r)=0$ (or the advantage is $0$ for every member), so the group has nothing to say. Fix: dynamic sampling to drop those groups, easier prompts early, or more diverse sampling (temperature).

Length inflation. Symptom: mean completion length rises steadily while accuracy stalls; per-token loss looks healthy. Cause: the $1/|o_i|$ normalisation makes long wrong answers cheap. Fix: Dr. GRPO’s no-length-normalisation variant, DAPO’s overlong penalty, or an explicit length term in the reward.

Reward hacking of the verifier. Symptom: verifier reward approaches 1 while held-out accuracy does not, and the outputs acquire a suspicious format. Cause: the reward is exact only for the property it checks — regex-shaped answer dumps, test-case overfitting, answer guessing. Fix: multiple checkers, held-out verification, and reading the outputs. The general statement is Amodei et al. 2016 (arXiv:1606.06565); the measurement framework is Gao et al. 2022 (arXiv:2210.10760).

Spurious rewards still “work”. Symptom: a run with deliberately random or incorrect rewards still improves benchmark scores. Cause: RLVR changes output format, length and entropy, and some benchmark gains come from those, not from reasoning. This is the most important cautionary result in the area (Spurious Rewards, arXiv:2506.10947). Fix: always report a format-ablated control.

Off-policy drift from engine mismatch. Symptom: instability that no learning-rate change fixes; the ratio’s tail grows across updates. Cause: vLLM/SGLang log-probabilities differ from the training engine’s, and num_iterations > 1 reuses stale rollouts. DeepSeek-V3.2 handles it with off-policy sequence masking and rollout IS correction. Fix: veRL rollout_correction, or keep num_iterations = 1.

Truncation mid-answer. Symptom: reward is low for a reason unrelated to reasoning; the model learns to emit a truncated answer. Cause: max_completion_length fires. Fix: DAPO’s overlong reward shaping, or raise the limit.

Truncated sampling breaking importance sampling. Symptom: instability when top-p/top-k are on. Cause: DeepSeek-V3.2: top-p/top-k “introduces a mismatch between the action spaces of ${\pi }_{\text{old}}$ and ${\pi }_{\theta }$”. Fix: keep the sampling mask and apply it during training.

Reading group-relative advantage as a value. Symptom: you conclude that a completion with ${\tilde{r}}_i=+2$ was good in absolute terms. Cause: the sign is only relative to its siblings. If every completion in the group was wrong, the best of them still gets a positive advantage. Fix: log both the raw reward and the advantage.

6. Using it in a project

Reach for GRPO when three things are true at once: the reward can be computed per complete response and is cheap; sampling $G$ completions per prompt is affordable (which is to say, the reward’s cost does not scale with $G$); and memory, not generation, is your binding constraint. That is the RLVR profile: maths with answer matching, code with unit tests, structured extraction with a schema checker, tool use with a deterministic success condition.

Do not reach for GRPO when:

• Your reward is dense or shaped and you need per-step credit. GRPO’s advantage is constant across a whole completion; turn-level or step-level signal is invisible to it. Use PPO with a critic, or look at VAPO, which adds the value model back with fixes for exactly this.
• Your reward is expensive per sample. $G$ completions means $G$ reward calls. A large learned RM with a long forward pass changes the arithmetic sharply; a rule-based verifier does not.
• You cannot sample a group. $G=1$ is not GRPO. Online DPO-style or RLOO variants may fit better.
• Your preference data is offline and thin. Then you do not have an environment to sample from at all; SFT for format, DPO for preference, and only then consider online RL.
• Tasks with no verifier and no RM. Agentic tasks often have neither; you are back to a learned reward model, and GRPO’s advantage is only as good as that proxy.

The 2026 framing, stated plainly. GRPO, not PPO, is the frontier default for reasoning post-training. [as of 2026-10] DeepSeek-V3.2 (arXiv:2512.02556) titles §3.1 “Scaling GRPO” and adds four corrections to it (unbiased KL, off-policy sequence masking, keep routing for MoE, keep sampling mask); Qwen3’s §4.2 states that it used GRPO to update the model on its 3,995 query-verifier pairs; DAPO and GSPO are GRPO edits; TRL ships GRPOTrainer and no PPO trainer. The strongest PPO counter-current is VAPO. Both halves of that sentence have primary sources.

The attribution caveat, which interviewers increasingly probe. Yue et al. 2025 (arXiv:2504.13837) argues that RLVR sharpens pass@1 by concentrating on reasoning paths the base model already had, without raising pass@k for large $k$ — “redistribution, not capability”. ProRL (arXiv:2505.24864) argues the opposite: prolonged RL does expand the reasoning boundary. This is an open empirical dispute with primary sources on both sides; do not present either as settled.

7. How you would prove it worked

The compute-matched baseline. Do not compare GRPO against PPO at equal steps. Compare at equal generated tokens, or against best-of-$G$ sampling from the SFT model with the same total token budget. (example) If best-of-8 from SFT already gets 70% on your held-out set and GRPO reaches 72%, the interesting claim is about inference cost at deployment — GRPO’s policy is cheap to sample from once, best-of-8 is not — not about a capability jump. This is the baseline that makes the reward-curve claim honest.

pass@1 and pass@k, together. Report both. A method that raises pass@1 and leaves large-$k$ pass@k flat is redistributing probability mass; a method that raises both has added capability. The two curves are the whole argument in Section 6’s caveat, and they are cheap to compute with the same evaluation harness.

The std-normalisation ablation. Run scale_rewards=True versus False (Dr. GRPO) with everything else fixed. (example) The expected signatures: the True run’s length drifts up faster, and the False run’s reward curve is noisier early because it loses the per-prompt scale correction. Report the length at matched reward, not just the reward.

Zero-variance-group rate. Log the fraction of groups with $\mathrm{std}(r)=0$ per step. It is the diagnostic that explains a suspiciously flat gradient norm, and it is the number that tells you whether your prompt difficulty or your temperature is wrong.

Reporting template (example). “On GSM8K with a 1.5B policy we compared GRPO ($G=8$) against best-of-8 sampling from the same SFT checkpoint at matched generated tokens. Held-out pass@1 rose from 61% to 68%; large-$k$ pass@k was unchanged, so we describe the effect as decomposition/mass-shifting rather than a capability gain. Dropping std normalisation cut mean response length by 18% at equal reward.”

8. Interview drill

“What exactly does GRPO change versus PPO?” Strong answer: it replaces the learned critic and GAE with a baseline computed from a group of $G$ samples of the same prompt: ${\hat{A}}_{i,t}=(r_i-\mathrm{mean}(r))/\mathrm{std}(r)$, constant over $t$. That removes $\gamma$, $\lambda$, the value model and its optimizer state; it keeps the ratio, the clip, the min, the reference model and the sampling loop. It is PPO minus the critic, not minus the RL. Common wrong answer: “GRPO has no reward model.” Often true in practice for RLVR, but it is not what defines the algorithm: GRPO works with any scalar reward, including a learned RM.

“Why is the group mean a valid baseline?” Strong answer: a baseline may be any function that does not depend on the action — here, any function of the prompt and the group. Because ${\mathbb{E}}_{a\sim \pi }[A^{\pi }(s,a)]=0$, subtracting it leaves the expected policy gradient unchanged and reduces variance. The group mean estimates exactly the quantity the baseline should be: the policy’s average reward on this prompt. Common wrong answer: “because it is an unbiased estimate of the value function.” It is a sample estimate with its own variance, and it is computed from the same samples the gradient uses — that is fine for a baseline but it is not an unbiased value estimate in general.

“How much memory does removing the critic save?” Strong answer: an actor-sized trainable model plus its optimizer state — for a 7B model at bf16 with fp32 Adam moments, roughly 12 bytes per parameter versus 2 for an inference-only copy, so ~84 GB versus ~14 GB in a stated configuration. Derive it rather than quoting it, and state that ZeRO sharding and offload change the total. Common wrong answer: “it saves half the memory” without saying what is being counted, or forgetting the optimizer state, which is the largest term.

“What does GRPO cost instead?” Strong answer: $G\times$ the rollout tokens per prompt. Since rollout dominates wall clock (OpenRLHF’s self-report is 80%), the trade is memory down, generation up. It is a bad trade when the reward is expensive per sample or when generation, not memory, is the bottleneck. Common wrong answer: “GRPO is strictly cheaper.” It is cheaper in memory and critic-fitting difficulty; it is more expensive in generation.

“Why did it matter for reasoning models?” Strong answer: RLVR gives a cheap, near-binary outcome reward, and a binary outcome over a group is exactly what GRPO’s advantage needs — with $r_i\in \{0,1\}$, $\mathrm{mean}(r)$ is the model’s own pass rate, so the advantage literally means “did this sample beat my pass rate”. Meanwhile no critic has to be fit to predict the return of a 32k-token chain of thought. DeepSeek-R1 showed this produces emergent long reasoning; DeepSeek-V3.2 and Qwen3 kept it. Follow-up: “is that causal?” → no, and say why: Dr. GRPO shows part of the effect is base-model and prompt-template artefact, and Yue et al. argue RLVR redistributes rather than extends capability.

“What are GRPO’s known biases?” Strong answer: two, both in Dr. GRPO’s critique. The $1/|o_i|$ per-sequence normalisation dilutes the penalty on long wrong answers, so the policy drifts longer. The local $\mathrm{std}(r)$ normalisation inflates advantages for prompts with low reward variance — including all-wrong groups — and is a biased estimator; REINFORCE++ proposes global advantage normalisation instead. Common wrong answer: “GRPO is unbiased because the group mean is a baseline.” The baseline is fine; the normalisation choices are where the bias lives.

“Why not just set $G$ very large?” Strong answer: the baseline’s variance falls like $1/G$ while the rollout cost grows like $G$; the returns flatten quickly, and diversity (temperature) matters more than group size once $G$ is moderately large. Also, large $G$ does not fix zero-variance groups when the prompt is too hard for the current policy. Common wrong answer: “$G$ is like batch size, so bigger is always better.” It is a per-prompt estimator, not a batch.

Next: DPO — remove the RL loop entirely: the same preference data, optimised with a pairwise classification loss and no sampling at all.