RL Frameworks

TL;DR — An RLHF run is not a training script. It is three or four models with different resource shapes, two execution regimes (bandwidth-bound autoregressive generation and compute-bound training), and a rendezvous problem between them, all on the same set of GPUs. This chapter is about the systems half: how much memory each role costs, why colocating them changes the arithmetic from a sum to a max, why generation eats the wall clock, and how weights physically move from the trainer to the rollout engine (NCCL broadcast, CUDA IPC, shared memory, or disk). TRL no longer ships PPOTrainer [as of 2026-10]; the frameworks that implement PPO/GRPO at scale are OpenRLHF, veRL, SkyRL and DeepSpeed-Chat. Reach for one of them only when the algorithm you need cannot be expressed as a single-controller loop.

1. Why this exists

You can write PPO for LLMs in 60 lines. The 60 lines are not the problem. The problem is that the loop needs four models in memory at once — an actor, a critic, a reward model and a reference — plus an inference engine generating completions from the actor, and each of those wants the GPU for a different reason at a different time.

Concretely, the first naive version fails like this. You load the actor in transformers, generate a batch with model.generate(), score it with the reward model, compute GAE, and take a gradient step. Two things happen. First, the generation step leaves the GPU almost idle: autoregressive decoding reads the entire model from HBM to produce one token per sequence, so the arithmetic intensity of that phase is tiny compared to the backward pass. Second, the moment you try to keep a critic alongside the actor you discover you are holding two trainable 7B models with Adam state — roughly 224 GB derived below — and the run does not start at all on the hardware you have.

No amount of PPO tuning fixes either failure. In Part 03 the progression PPO -> GRPO -> DPO is a sequence of algorithmic cost reductions; the framework layer is a second, orthogonal set of systems cost reductions:

• Placement — put the models that are never updated on the same GPUs as the models that are, and time-share instead of duplicating.
• Overlap — run generation and training at the same time instead of alternating, and pay for the resulting off-policyness with an importance correction.
• Weight synchronisation — get the actor’s new weights into the inference engine without writing a checkpoint to disk and reading it back.

The orchestration layer that makes this possible is almost always Ray, which is why veRL, OpenRLHF and SkyRL all put a Ray actor in front of every GPU. The Ray chapter covers the core primitives (actors, placement groups, the object store); this chapter only uses them.

The interview framing. “Which RL framework should we use?” is the wrong question. The right one is: what is my binding constraint — memory, wall clock, or algorithmic control — and which placement decision does it force? Everything below is that question with numbers attached.

2. How it works

2.1 The roles are four different resource shapes

Before any placement decision, name what each resident object is and what it wants. This table is the whole chapter compressed:

The symbols are the ones the PPO chapter and the GRPO chapter use: ${\pi }_{\theta }$ is the policy being trained, $V_{\psi }$ the learned value baseline, $r_{\varphi }$ the reward model whose scalar output is the training signal, and ${\pi }_{\text{ref}}$ the frozen SFT model that appears in the KL denominator. Nothing here is new mathematics; what is new is that each row has a different memory and compute profile, and the framework’s job is to schedule those profiles against each other.

The memory of each role is one multiplication:

$$M_{\text{role}}=P\cdot b_{\text{role}}$$

• $P$ — the parameter count of that model. For PPO with a critic this is usually the same $P$ for the actor, the critic and the reference, which is exactly why the critic is so expensive: it is not a small head, it is a second actor-sized network.
• $b_{\text{role}}$ — bytes per parameter for this role’s precision mode. Inference-only is easy: 2 bytes for a bf16 weight, 1 for fp8. Trainable is not, and this is the number people get wrong.

$b_{\text{train}}$ decomposes as:

$$b_{\text{train}}=\underset{\text{bf16 weight}}{\underbrace{2}}+\underset{\text{bf16 grad}}{\underbrace{2}}+\underset{\text{fp32 master}}{\underbrace{4}}+\underset{m}{\underbrace{4}}+\underset{v}{\underbrace{4}}=16\text{bytes}$$

Every term is load-bearing:

• bf16 weight (2) — the tensor used in the forward and backward pass. bf16 is the compute format because tensor cores run it at full rate.
• bf16 gradient (2) — accumulated per parameter during backprop.
• fp32 master weight (4) — the authoritative copy that the optimizer updates. Remove it and you update the bf16 weight directly: with a learning rate of $3\times 10^{-6}$, the update $-\eta g$ is far below bf16’s resolution near any weight of magnitude ~1, so the parameter rounds to itself and stops moving. This is the single most common “the loss is completely flat” bug in from-scratch mixed-precision training, and it is why the master copy exists.
• Adam $m$ and $v$ (4 + 4) — first and second moment buffers, both fp32 because $v$ is an estimate of a squared gradient and bf16’s 8 bits of mantissa destroys it. Remove $m$ and you have SGD with momentum only; remove $v$ and you have RMSProp, which changes the effective step size per parameter and silently invalidates any learning rate you tuned.

Reconciling two numbers in this book. The PPO chapter quotes $12$ bytes/param by dropping the fp32 master weight; the DeepSpeed/veRL convention here keeps it and gives $16$. Both are correct for a stated configuration, and a ZeRO-3 or FSDP run that shards the master copy still pays for it somewhere. The same choice moves every total in this part of the book: the GRPO chapter inherits PPO’s $12$-byte convention and quotes 196 GB for PPO and 98 GB for GRPO, while Section 4.2 below uses $16$ bytes and quotes 252 GB and 140 GB. Two differences, not one. PPO’s $196\to 252$ is the convention alone: $4$ extra bytes on each of its two trainable copies is $56$ GB. GRPO’s $98\to 140$ is the convention on its single trainable copy ($28$ GB) plus $14$ GB because the GRPO chapter assumes the reward is a free verifier while Section 4.2 counts an inference-only reward model. State which convention and which reward setup you are using before quoting a total; a 33% difference in a memory budget is the difference between fitting and not fitting.

2.2 Why colocating changes the arithmetic

Here is the calculation that decides your hardware. Let

$$M_{\text{GPU}}=\frac{P\cdot b_{\text{train}}\cdot n_{\text{train}}+P\cdot b_{\text{infer}}\cdot n_{\text{infer}}}{W}+A$$

• $n_{\text{train}}$ — how many trainable copies are resident. PPO: 2 (actor and critic). GRPO: 1 (actor only). DPO: 1, plus one inference-only reference.
• $n_{\text{infer}}$ — how many inference-only copies are resident. Usually 2 (reward model and reference), and 0 if the reward is a verifier and the reference log-probs were precomputed.
• $W$ — the number of GPUs the models are sharded across. This is where FSDP/ZeRO earns its place: without sharding, every GPU holds a full copy and the division does not happen.
• $A$ — the activation peak. This is the term that FSDP must recompute or offload, and it scales with tokens in flight (batch x sequence length) rather than with parameter count, so it is the term that explodes when you raise response_length or the group size $G$.

For a 7B model (example):

That does not fit on one 80 GB device, and it is why the very first thing an RL framework does is shard each role across a group of GPUs and then place the groups. Two placements are possible, and the names differ per framework but the idea is identical:

Disaggregated. Each role gets its own GPUs. The actor trains on 8 GPUs, the rollout engine owns 4 more, the reward model 1, the reference 1. Total is the sum of the roles’ footprints, and you need W_train + W_rollout GPUs, but generation and training can genuinely overlap.

Colocated (hybrid engine). The same GPUs run several roles by time-slicing. The actor’s weights are the rollout engine’s weights — one copy, not two — and the reward model and reference are loaded for their phase and freed, or kept in a slept state. The peak is now the max over phases, not the sum:

$$M_{\text{peak, colocated}}=\underset{\text{phase}}{\max }\left(M_{\text{phase}}\right)+A_{\text{phase}}$$

The three phases are rollout (weights + KV cache), scoring (reward model + reference), and update (weights + gradients + Adam + activations). Each phase has a different binding term, which is why colocated runs are so sensitive to ordering: veRL’s own docs note that the rollout engine is created last “so that vllm can have a better estimation of kv cache memory” — under colocation, KV-cache sizing is a first-class ordering constraint rather than an afterthought.

What colocation buys and costs, concretely:

• Buys: no duplicated weight copy (14 GB for 7B (example), proportional to $P$ for anything larger), no cross-node weight transfer needed if the engine is on the same node, and it works on a small GPU count. For a 7B PPO run, disaggregated needs something like 8 training GPUs plus a 2-4 GPU rollout pool; colocated needs 8 total (example).
• Costs: generation and training cannot overlap. The node alternates generate -> sync -> train -> sync, and it is idle at each boundary while the other phase has nothing to do. That idle time is the dominant inefficiency in synchronous colocated RL, and it is the reason every framework shipped an asynchronous mode in 2025-2026.

The order of magnitude that matters. Colocation is a memory optimisation that creates a latency problem. Async is the latency fix that reintroduces an off-policyness problem. The last two years of RL-systems work are this loop; see Section 2.6.

2.3 The rollout/train split, and why generation dominates wall time

OpenRLHF’s README states the design premise flatly: “RLHF training spends 80% of the time on sample generation” (a project self-report, not a controlled measurement). The interesting part is why that is true, because on raw FLOPs it should not be. Forward-only training compute is $2ND$ FLOPs for $N$ parameters and $D$ tokens; forward plus backward is about $6ND$; and PPO takes $K$ epochs over the same rollout batch, so training compute should exceed generation compute. It does not, for two reasons.

Reason one: decoding is memory-bandwidth-bound. To emit one token, the engine reads every weight from HBM and does almost no arithmetic per byte. The per-step time at batch 1 is approximately

$$t_{\text{decode}}\approx \frac{P\cdot b_{\text{infer}}+\text{KV bytes}}{B{W}_{\text{HBM}}}$$

• $P\cdot b_{\text{infer}}$ — the weight bytes that must be streamed through the SMs. For 7B in bf16 that is 14 GB.
• $\text{KV bytes}$ — the cache that must also be read; this grows with in-flight sequence length and is derived in the KV Cache chapter. It is why long-context RL is disproportionately a rollout-side cost.
• $B{W}_{\text{HBM}}$ — memory bandwidth. On an H100 this is roughly 3.35 TB/s.

For 7B bf16 at batch 1, the weight term alone gives $14\text{GB}/3.35\text{TB/s}\approx 4.2$ ms per token (example) — where a training forward-backward over the same token takes a small fraction of that. Batching amortises it: one weight sweep serves every sequence in the batch, which is exactly what continuous batching and PagedAttention (vLLM) exist to exploit. But rollout batches are bounded by KV-cache memory and by the variance in response length, so you rarely reach the batch size at which decode becomes compute-bound. Achieved MFU during generation stays a small fraction of what the backward pass reaches.

Reason two: responses are long and every one of them must be fresh. RLHF is on-policy: the gradient’s expectation is over ${\pi }_{\theta }$, so a stale completion is a biased sample. A response is 256 to 32,768 tokens, and response_length multiplies the rollout term directly. GRPO multiplies it again by the group size $G$, because it samples $G$ completions per prompt to build a baseline with no critic.

So the two halves of an RL step have opposite hardware appetites, and that is the sentence to remember:

• Rollout wants a large KV cache, high concurrency, low precision, and tolerates latency. It is bandwidth-bound and embarrassingly parallel.
• Training wants activation memory and optimizer state. It is compute-bound and tightly coupled by collectives.

Any framework that puts them on the same device is arbitrating between two incompatible batch shapes, and any framework that separates them must pay a weight-transfer tax every iteration (Section 2.5).

2.4 The actor topology

Every modern framework lands on the same shape: one lightweight controller process, one group of Ray actors per role, and gang-scheduled placement so a collective never has a missing rank. In veRL the controller is the RayPPOTrainer, which “runs on the driver process on a single CPU/GPU node (default is CPU)”; roles are WorkerGroups of Ray actors named actor_rollout, critic, ref and rm. OpenRLHF’s equivalent is RayActorGroup, which builds one actor per rank; SkyRL’s is PPORayActorGroup plus a Controller.

flowchart TB
    D["Driver (CPU): RayPPOTrainer<br/>build batches, compute advantages, orchestrate"]
    subgraph PG["Ray placement group - one bundle per rank, STRICT_PACK within a node"]
        AR["actor_rollout WorkerGroup"]
        CR["critic WorkerGroup"]
        RF["ref WorkerGroup"]
        RM["rm WorkerGroup"]
    end
    AR --> ENG["Rollout engine actors<br/>vLLM / SGLang"]
    AR --> ENV["Environment / verifier<br/>reward function, tools, unit tests"]
    D --> AR
    D --> CR
    D --> RF
    D --> RM
    AR -. "weights every iteration" .-> ENG

Named pieces, because each one is a place a real run breaks:

• The driver. It orchestrates, computes advantages in some backends, and moves data between role groups. It is a CPU-side serialization point: veRL runs compute_advantage on the driver, so as your batch grows, driver time grows while every GPU waits.
• The actor_rollout group. The name is the design. The actor and the rollout engine are the same weights, and the framework’s choice of hybrid_engine: true decides whether they are also the same GPUs.
• The environment / verifier. In an RLVR run the “reward model” is a function, not a network — veRL’s reward-model block defaults to disabled and its math examples use rule-based rewards. A CPU-side verifier is not a GPU role, but it is a latency role: a slow verifier or a flaky tool stalls the rollout loop the same way a slow model would.
• The bundles. Under Ray, a bundle must fit on a single node, so a 4-GPU tensor-parallel group is modelled as one {"GPU": 4} bundle with STRICT_PACK; the default PACK is soft and will scatter your collective across nodes. Details, including the actor-CPU hang, are in the Ray chapter.

2.5 Weight synchronisation: four strategies, four cost profiles

When the rollout engine is not literally the training model, its weights are stale the moment the optimizer steps. The next rollout must see the new weights, so every framework implements some form of trainer -> engine transfer. The cost is

$$T_{\text{sync}}=\frac{P\cdot b_{\text{wire}}\cdot r}{B{W}_{\text{eff}}}+T_{\text{reshard}}$$

• $P\cdot b_{\text{wire}}$ — the payload in bytes. $b_{\text{wire}}$ is 2 for bf16, 1 for fp8; quantising the transfer is a real option (SkyRL ships an fp8/ path) and trades a small numerical error for a 2x cheaper sync.
• $r$ — the replication factor: how many copies cross the wire. A tree broadcast is $r=1$ in the sense that the payload is read once at the root and forwarded; a naive point-to-point send to each engine replica is $r=\text{replicas}$. This is the difference between a broadcast and a loop of sends, and it is the difference between 30 ms and 300 ms.
• $B{W}_{\text{eff}}$ — effective bandwidth of whatever transport you chose. The hierarchy is NVLink > PCIe > cross-node RDMA > Ethernet.
• $T_{\text{reshard}}$ — the cost of rearranging the tensors from the trainer’s layout to the engine’s. This is the term people forget. FSDP shards a parameter across ranks by flat slices; vLLM shards an attention projection across ranks by head. The same weights must be gathered, re-split, and re-laid-out, and for a model with many small tensors this is thousands of round trips. It is the entire motivation for veRL’s 3D-HybridEngine (HybridFlow, arXiv:2409.19256) and for its bucketed transfer path, whose docstring is literally “Bucketed weight transfer via ZMQ + IPC (or shared memory fallback).”

The rule for choosing: does an NCCL communicator span both sides? If yes, broadcast. If no but they share a node, CUDA IPC or shared memory. If neither, you are shipping bytes through storage or the network, and the sync frequency is now a first-class tuning knob — which is why veRL’s async trainer exposes parameter_sync_step, and why SkyRL ships a benchmark specifically for the delta path.

Worked numbers (example), 7B model, 14 GB payload: on-node broadcast at an effective 450 GB/s is about 31 ms; over PCIe Gen5 at ~64 GB/s about 0.22 s; over 100 GbE (12.5 GB/s) about 1.1 s; over 400 GbE (50 GB/s) about 0.28 s. At a training step of a few seconds, a 31 ms sync is noise and a 1.1 s sync is a 30% overhead. That is why 100 GbE clusters are fine for torchrun and painful for disaggregated RL.

2.6 Synchronous, colocated-async, and fully-async

veRL’s current config exposes the three modes directly as trainer.v1.trainer_mode with values sync, colocate_async and separate_async [as of 2026-10]. They are the clean statement of the trade:

flowchart LR
    subgraph S["sync (colocated, strictly on-policy)"]
      S1["generate"] --> S2["sync weights"]
      S2 --> S3["train"]
      S3 --> S1
    end
    subgraph A["separate_async (overlapped, partially off-policy)"]
      A1["rollout pool: generating batch k+1"] 
      A2["trainer: updating on batch k"]
      A1 -. "weights every parameter_sync_step" .-> A2
    end

• sync. Generation and training alternate. Weights are always current, so the data is exactly on-policy and the importance ratio in the PPO/GRPO objective is 1 at the start of each update. The price is idle GPU time at every phase boundary.
• colocate_async. Generation and training share the GPUs but overlap through a queue. Samples in flight may have been produced by an older policy version, so the objective’s ratio is no longer 1 and must do real work.
• separate_async. A dedicated rollout pool keeps generating while the trainer updates. This is the throughput-maximising configuration and the most off-policy. veRL bounds the staleness with trainer.v1.sampler.max_off_policy_threshold (default 8: “maximum number of model versions that trajectory can span”) and max_off_policy_strategy: drop | wait.

The correctness cost of asynchrony is not hypothetical, and the corrections all share one name: importance sampling. When the data comes from ${\pi }_{{\theta }_{\text{old}}}$ and the gradient is for ${\pi }_{\theta }$, every term must be multiplied by $\rho ={\pi }_{\theta }/{\pi }_{{\theta }_{\text{old}}}$, and $\rho$ is heavy-tailed. Three countermeasures appear in the frameworks, and each is a symptom of a different source of off-policyness:

1. Minibatching and PPO epochs — the original, intended source. Handled by the clip and by lam/K.
2. Training/inference engine mismatch — vLLM’s log-probs do not exactly match FSDP’s. veRL exposes algorithm.rollout_correction.rollout_is with rollout_is_threshold: 2.0 to truncate the ratios, and requires actor_rollout_ref.rollout.calculate_log_probs=True to have them at all. DeepSeek-V3.2 states the problem: “inference frameworks used for efficient data generation are often highly optimized, which may differ in implementation details from training frameworks. Such training-inference inconsistency further exacerbates the degree of off-policyness.” (arXiv:2512.02556)
3. Sampling truncation — top-p/top-k during rollout changes the action space, which breaks the ratio’s denominator outright. DeepSeek-V3.2 fixes it by recording the truncation mask during sampling and re-applying it during training (“Keep Sampling Mask”), so both policies share an action subspace.

3. Implementation

3.1 A real veRL launch, every parameter explained

veRL runs on Hydra overrides on top of verl/trainer/config/*.yaml. The config below is a single-node 8-GPU GRPO-on-verifiable-rewards run. Every key was checked against verl/trainer/config/ppo_trainer.yaml and verl/trainer/config/rollout/rollout.yaml on verl-project/verl main [as of 2026-10]:

# veRL, single node, 8 GPUs, GRPO against a rule-based reward.
python -m verl.trainer.main_ppo \
  algorithm.adv_estimator=grpo algorithm.gamma=1.0 algorithm.lam=1.0 \
  algorithm.use_kl_in_reward=False algorithm.kl_penalty=low_var_kl \
  algorithm.kl_ctrl.type=fixed algorithm.kl_ctrl.kl_coef=0.001 \
  algorithm.rollout_correction.rollout_is=token \
  algorithm.rollout_correction.rollout_is_threshold=2.0 \
  data.train_files=$DATA/train.parquet data.val_files=$DATA/test.parquet \
  data.max_prompt_length=1024 data.max_response_length=8192 data.train_batch_size=256 \
  actor_rollout_ref.model.path=Qwen/Qwen2.5-7B-Instruct \
  actor_rollout_ref.actor.optim.lr=1e-6 actor_rollout_ref.actor.ppo_mini_batch_size=64 \
  actor_rollout_ref.actor.use_kl_loss=True \
  actor_rollout_ref.actor.fsdp_config.param_offload=False \
  actor_rollout_ref.actor.fsdp_config.optimizer_offload=False \
  actor_rollout_ref.rollout.name=vllm actor_rollout_ref.rollout.mode=async \
  actor_rollout_ref.rollout.tensor_model_parallel_size=2 \
  actor_rollout_ref.rollout.gpu_memory_utilization=0.5 actor_rollout_ref.rollout.n=8 \
  actor_rollout_ref.rollout.free_cache_engine=True \
  actor_rollout_ref.rollout.enable_prefix_caching=True \
  actor_rollout_ref.rollout.calculate_log_probs=True \
  actor_rollout_ref.rollout.checkpoint_engine.backend=nccl \
  actor_rollout_ref.rollout.checkpoint_engine.update_weights_bucket_megabytes=2048 \
  actor_rollout_ref.hybrid_engine=true trainer.v1.trainer_mode=colocate_async \
  trainer.v1.sampler.max_off_policy_threshold=8 \
  trainer.n_gpus_per_node=8 trainer.nnodes=1

Why each of the load-bearing knobs is where it is:

• algorithm.adv_estimator=grpo — selects group-relative advantages and removes the critic entirely (GRPO: group mean as baseline). Setting gae instead re-enables PPO and the critic; reinforce_plus_plus_baseline, rloo, rloo_vectorized, grpo_vectorized are the other options. Default is gae.
• algorithm.lam=1.0 — the GAE bias-variance dial. With a critic removed, there is nothing to bootstrap from, so lam is inert; it is quoted here because it matters the instant you switch to gae, where 1.0 means “use the empirical return minus a baseline” (unbiased, high variance) and 0.95 is the verified N+ setting.
• algorithm.use_kl_in_reward=False — where the KL penalty lives. False (the documented default) means the KL is a loss term rather than a per-token reward subtraction. This is a 2025-2026 convention change: modern recipes put the KL in the objective, and for verifiable math domains DeepSeek-V3.2 notes it can be weakened or dropped.
• algorithm.kl_penalty=low_var_kl — which estimator. kl is the naive single-sample $\log {\pi }_{\theta }-\log {\pi }_{\text{ref}}$, unbiased but high-variance; low_var_kl is the k3 estimator, which is non-negative and lower-variance. abs, mse and full exist; changing this changes the gradient’s noise, not its expectation.
• algorithm.kl_ctrl.kl_coef=0.001 — the coefficient $\beta$. Note the churn: the current default is 0.001, while older docs and examples carry 0.005, and the PPO chapter’s N+ value is 0.05. These are not comparable across codebases because $\beta$ is entangled with the reward scale and normalisation; changing it changes how far the policy may drift per unit of reward.
• actor_rollout_ref.actor.ppo_mini_batch_size=64 — the minibatch for each gradient step inside the rollout batch. Smaller means more steps and noisier gradients; larger means fewer, lower-variance steps but more memory.
• actor_rollout_ref.actor.use_kl_loss=True — attaches the reference model to the actor group so its log-probs are available for the loss. This is the thing that makes ${\pi }_{\text{ref}}$ resident; turning it off is the cheapest way to save an inference copy.
• actor_rollout_ref.rollout.tensor_model_parallel_size=2 — how many GPUs one engine instance spans. TP is what forces the reshard: the trainer’s FSDP layout and the engine’s TP layout are different splits of the same tensors.
• actor_rollout_ref.rollout.gpu_memory_utilization=0.5 — the fraction of device memory vLLM may use for weights plus KV cache. Under colocation this is the single most dangerous number in the config: it is competing with the optimizer state and activations of the training phase for the same HBM. Too high and training OOMs; too low and the KV cache cannot hold enough concurrent sequences, so rollout throughput collapses and the GPU sits idle.
• actor_rollout_ref.rollout.n=8 — the group size $G$: completions sampled per prompt. It multiplies rollout compute and KV pressure linearly and is what gives GRPO its baseline. Verified at this path — the config file uses actor_rollout_ref.rollout.n, so a blog post or doc that writes actor_rollout.ref.rollout.n has a typo.
• actor_rollout_ref.rollout.free_cache_engine=True — free the KV cache between phases. Under colocation this is what makes the memory budget a max rather than a sum; leaving it off defeats the point of colocating.
• actor_rollout_ref.rollout.checkpoint_engine.backend=nccl — which weight transport (Section 2.5). Options are naive, nccl, nixl, hccl; naive is the disk-shaped fallback. update_weights_bucket_megabytes=2048 is the bucket size for batched updates; it exists to amortise the per-tensor overhead of many small transfers.
• actor_rollout_ref.hybrid_engine=true — the colocation switch. The config’s own comment is explicit: “Set to False only with the v1 separate_async trainer… All other trainers require True.”
• trainer.v1.trainer_mode=colocate_async — sync for strictly on-policy, colocate_async for shared-GPU overlap, separate_async for a dedicated rollout pool.
• algorithm.rollout_correction.rollout_is=token with rollout_is_threshold=2.0 — truncate the importance ratios at 2.0 so a handful of badly-mismatched tokens cannot dominate the update. Requires calculate_log_probs=True. If you run two engines, this is not optional bookkeeping.

Flag paths move between releases faster than any book. Treat this as a skeleton, and check verl/trainer/config/ in the checkout you are actually using before you paste it. The two failure modes are an override that raises (“Could not override”) and, worse, an override that is silently ignored because the key moved under a different parent.

3.2 The same run in OpenRLHF

OpenRLHF expresses the same decisions as flat CLI flags on python -m openrlhf.cli.train_ppo_ray. The mapping, so the two frameworks stop looking like different universes [as of 2026-10]:

The README’s own mode table is worth internalising: Hybrid Engine (--train.colocate_all + --vllm.enable_sleep + --ds.enable_sleep) is “strictly on-policy, serial generate -> train”; Async (--train.async_enable) overlaps generation and training, with “larger queue = more off-policy”; Async + Partial Rollout adds --train.partial_rollout_enable, which lets vLLM pause and resume so in-flight samples may mix old and new weights. That last mode is the honest name for what asynchronous RL actually is.

One Ray idiom from OpenRLHF worth stealing regardless of framework: in async_run_method_batch it slices the batch per actor and calls ray.put(chunk) once per chunk before dispatching, with the comment “This avoids transferring the full batch to every node (critical at scale).” Passing a large object directly to N remote calls transfers it N times.

3.3 What actually differs between them: the abstraction boundary

The frameworks do not differ much in algorithm list; they differ in what they make you think about. TRL makes you think about a dataset and a config dataclass. OpenRLHF makes you think about which roles share a GPU. veRL makes you think about a dataflow graph and worker groups. SkyRL makes you think about an environment, which is why it is the framework to cite for agentic RL: the same trainer drives single-turn math, multi-turn SQL tool use, and long-horizon terminal agents, and its Generator interface is a single generate() method.

4. Numbers and trade-offs

4.1 The framework landscape [as of 2026-10]

Two claims to state precisely, because both are easy to get wrong:

• TRL is not the PPO reference implementation any more. On main, the trainer exports are exactly SFTTrainer, DPOTrainer, KTOTrainer, GRPOTrainer, RLOOTrainer, RewardTrainer, DistillationTrainer — there is no ppo_trainer.py and no PPOTrainer in the release. Verify with one command: python -c "import trl; print(hasattr(trl, 'PPOTrainer'))".
• DeepSpeed-Chat is algorithmically superseded, not abandoned. Its repo is not archived and its README contains no deprecation banner, so do not claim Microsoft said it is dead. What is true and checkable is that it offers PPO only, with no GRPO/DAPO/RLVR path and no vLLM/SGLang rollout, and that its recent commits in that path are maintenance. Its lasting contribution is the Hybrid Engine idea — the actor and the inference engine time-sharing GPUs — which reappears under three different names in the other three frameworks.

RLlib, for completeness. Ray’s own RL library has no first-class LLM support [as of 2026-10]: there is no LLM/RLHF path in its docs or package. The LLM RL story in the Ray ecosystem lives entirely in veRL, OpenRLHF and SkyRL. Mentioning RLlib as an option for LLM post-training is a red flag in an interview.

4.2 Memory, derived for two scales

Using $b_{\text{train}}=16$ and $b_{\text{infer}}=2$, and counting only weights, gradients and optimizer state (activations excluded):

The reading is not “70B is impossible” — FSDP/ZeRO-3 shards it, and 2520 GB across 32 x 80 GB devices is 79 GB/device before activations, which is why 70B-class PPO runs use CPU offload or much larger worlds. The reading is: removing the critic removes roughly 45% of the static memory bill (16 of 36 bytes/param) and one full backward pass, which is the systems case for GRPO that the GRPO chapter makes algorithmically.

4.3 Where the wall clock goes

For a synchronous colocated step, the timeline is fixed and the amounts are recognisable:

Two structural facts fall out. First, increasing $G$ or response_length costs rollout time roughly linearly and training time roughly linearly, but rollout is the term with the worse hardware efficiency, so it dominates the total first. Second, the sync cost is proportional to model size and independent of batch size, so it hurts small models relatively little and large models a lot; that is why large-scale disaggregated runs invest in bucketed, quantised, and delta-based transfer and small ones do not bother.

4.4 Which axis a placement decision moves

5. Pitfalls

Rollout/train imbalance leaves GPUs idle. Symptom: step time is dominated by generation, MFU during the update phase is decent, and total GPU utilisation sits at 40-60%. In a synchronous colocated run this is structural: whichever phase is running, the other half of the machine is waiting. Diagnosis: instrument the per-phase timers (OpenRLHF exposes timing/generation, timing/make_experience, timing/ppo_train, timing/broadcast, timing/step_total — if the sum of the parts is well below the step time, you are measuring wasted boundary time, not slow compute). Fix: async mode, more rollout replicas, or a smaller $G$.

Weight-sync stalls. Symptom: a fixed multi-second block in every step that does not shrink when you shrink the batch, or a hang immediately after the trainer logs “broadcast”. A hang almost always means one rank is not in the collective — typically a mismatch between the trainer’s world size and the engine’s TP x replicas, or a placement that put one bundle on a different node than the communicator expects. Fix: check the communicator membership before blaming the network, and prefer a transport whose reachability you have actually verified (Section 2.5).

Checkpoint-engine / resharding mismatch. Symptom: the first few steps look fine, then the policy degrades into repetition or gibberish while the loss stays plausible; or the mean importance ratio $\rho$ drifts far above 1 and the clip fraction jumps to ~1. This is the engine running on partially updated or badly resharded weights — the trainer’s tensors were laid out for FSDP and the engine expected TP heads. Fix: compare per-token log-probs from the rollout engine against the trainer’s on a fixed batch; they should agree to within numerical noise. If they do not, the weights are not what you think they are, and rollout_is is masking a bug rather than correcting a distribution shift.

Version churn breaking configs. Symptom: Could not override 'algorithm.kl_coef' at launch, or worse, the run starts and silently uses a different default than the one in your notes. This is not hypothetical: kl_coef’s default moved from 0.005 in the docs to 0.001 in the current config; veRL moved from volcengine/verl to verl-project/verl in 2026-01; Ray removed RayServeHandle at 2.10 and changed max_ongoing_requests’s default at 2.32.0. Fix: pin the framework version in your launch script, print the resolved config at startup (Hydra will do it), and treat any config snippet older than a few months as a sketch.

Placement-group deadlock. Symptom: hangs before the first step, with no error. Two rungs of the same trap: the driver holds the resource it is waiting for, or a child task does not inherit the parent’s placement group and gets scheduled elsewhere — so a rank inside a STRICT_PACK group blocks forever on a helper that will never join. Diagnosis order: ray status, ray list placement-groups --detail, ray list actors (the actor’s required_resources should show the CPU_group_<pgid> pseudo-resource if it is really in the group). The Ray-specific detail is in the Ray chapter.

The actor-CPU trap. Symptom: a bundle of pure {"GPU": 1} with a default Ray actor hangs forever with “cannot schedule actor… requires 1 CPU”. Ray actors require 1 logical CPU at schedule time and release it immediately after, so the bundle must contain a CPU. Fix: set num_cpus explicitly on every actor, or add "CPU": 1 to the bundle.

Driver serialisation. Symptom: GPU utilisation drops as the batch grows while the driver’s CPU is pegged; step time scales with batch size faster than compute does. Advantages and reward aggregation run on the driver in some backends, so the single-controller design has a visible ceiling.

6. Using it in a project

Reach for a heavy RL framework only when the thing you are doing cannot be expressed as a single-controller training loop. The decision order:

1. Does SFT already pass the eval? If yes, stop. The RLHF overview makes the same point: alignment work is only worth starting when there is a measured behaviour gap that SFT data cannot close.
2. Do you have preference pairs and a fixed dataset? Then you want DPO or one of its relatives, not an RL framework. DPO is a classification loss over pairs with a cached reference log-prob: one process group, no rollout engine, no reward model, no critic, no placement problem. TRL’s DPOTrainer is the entire answer. Use an RL framework here and you have added three distributed systems to solve a problem that has none.
3. Single node, model fits, small scale? Then a single-controller trainer is fine. TRL’s GRPOTrainer (the fused LM head keeps the logits tensor from ever being materialised) or a small OpenRLHF run will do. Ray earns its place when the workload is heterogeneous, dynamic and stateful — several resource types, variable replica counts, long-lived engines. One model, one node, fixed topology is exactly the case where torchrun is simpler and stronger.
4. Do you need a rollout engine the trainer cannot be? The moment you want vLLM/SGLang throughput, multi-turn tool environments, or a policy that does not fit in the training process, you need the placement abstraction, and you are now choosing between OpenRLHF (Ray + DeepSpeed, easiest flag-level colocation story), veRL (the most flexible dataflow and the broadest algorithm list; the default choice for a new large-scale run), and SkyRL (choose it when the environment is the complex part — long-horizon agentic RL over tools).
5. Algorithmic control. If you are implementing a new estimator rather than running an existing one, veRL’s worker-group abstraction lets you express it as a dataflow change rather than a fork. If you are running a published recipe, check whether it ships a veRL recipe first — DAPO, GSPO, PRIME and the Dr. GRPO variants all do.

When NOT to use any of them, in one line. You have pairs and no environment: use DPO. You have one node and a small model: use TRL’s GRPOTrainer. SFT already passes the eval: use nothing. The framework is for the case where generation and training are genuinely two different jobs that must be scheduled against each other.

7. How you would prove it worked

A framework change is an engineering claim, so measure engineering quantities with a quality guardrail. Three experiments, in order:

1. The per-phase time breakdown, before and after. Log generation time, sync time, log-prob/scoring time, advantage time, and update time per step, and report the fractions, not just the total. The claim you are testing is “async overlap raised throughput”, and the falsifier is “the generation fraction did not move, the total just got noisier”. (example) A synchronous colocated run might show 72% generation / 6% sync / 22% train with 55% idle-boundary overhead in wall-clock terms; the same job in separate_async might show generation still at 72% of compute but now overlapped, for a 1.4x step-throughput gain. Report MFU for the update phase and tokens/s for rollout separately — a single aggregate hides which half changed.

2. The quality guardrail. Any throughput win from asynchrony is bought with off-policyness, so the ablation must show the reward curve did not degrade. Baseline: the same run in sync mode, same data, same seed where possible. Metric: held-out verifier accuracy (RLVR) or win-rate against the SFT model on a frozen preference set. A throughput gain with a reward-curve regression is a loss. (example) If async gives 1.4x step throughput but 2 points lower held-out accuracy, you have not won; if it holds accuracy and cuts wall clock 30%, you have.

3. The sync-cost ablation. Measure T_sync directly and check it against the derivation $Pbr/BW$. If measured time is much larger than the payload argument, the gap is $T_{\text{reshard}}$, and that is where the next optimization goes (bucketing, fp8, or eliminating the reshard by choosing a rollout backend whose layout matches the trainer’s). (example) A 7B model syncing in 400 ms where the bandwidth argument predicts 31 ms is a resharding problem, not a network problem, and swapping to a faster interconnect will not fix it.

Reporting template (example). “We moved the GRPO run from sync to separate_async with parameter_sync_step=4. Step throughput rose from 0.9 to 1.3 iterations/min at unchanged held-out verifier accuracy (example); the weight-sync block stayed under 4% of step time. The residual cost is staleness: the mean importance ratio rose from ~1.00 to ~1.02, well inside the 2.0 threshold, but it is the number we would watch first if reward began to regress.”

8. Interview drill

“Why does generation dominate RL training time when it is only a forward pass?” Strong answer: training is forward+backward (~3x the FLOPs) and PPO reuses the batch for $K$ epochs, so on FLOPs training should dominate; it does not, because autoregressive decoding is memory-bandwidth-bound. Each token requires sweeping the whole model from HBM with almost no arithmetic per byte, so achieved MFU is a fraction of the backward pass’s, and RLHF responses run to thousands of tokens and are multiplied by the group size $G$ in GRPO. OpenRLHF self-reports 80% of wall time on generation. Common wrong answer: “because the model has to generate a lot of tokens” — that restates the observation, and it is equally true of training tokens. The reason is the efficiency difference, not the token count.

“Why can’t you just use TRL’s PPOTrainer?” Strong answer: it no longer exists. On main, TRL exports SFT, DPO, KTO, GRPO, RLOO, Reward and Distillation trainers and no PPOTrainer; the deprecation was tracked in TRL issue #2016 and is absent as of v1.15.0. TRL is now a single-controller trainer library, which is a fine thing to be, but it is not where PPO lives. Use OpenRLHF, veRL, SkyRL or DeepSpeed-Chat for PPO. Common wrong answer: quoting TRL’s PPO hyperparameters from a 2023 tutorial. The tutorial is real and the API in it is gone.

“How do updated weights reach the rollout engine?” Strong answer: first the layout is reshaped — the trainer shards parameters by flat slice (FSDP) while the engine shards by attention head (TP), so the tensors must be gathered and re-split; that reshard is usually the expensive part. Then the payload moves by one of four transports: an NCCL broadcast when trainer and engine are in one communicator; CUDA IPC when they are colocated on a node; shared memory / bucketed ZMQ when they are separate processes with different layouts; and a checkpoint delta through shared storage when they are not NCCL-reachable at all. The cost is $Pbr/BW$ plus the reshard. Common wrong answer: “vLLM reloads the checkpoint from disk.” That works and is the fallback, but it is the slowest option and it is what the fast paths specifically exist to avoid.

“Colocated or disaggregated — which do you choose?” Strong answer: disaggregate when you can fill a dedicated rollout pool and want generation overlapped with training; colocate when GPUs are scarce, because it turns the memory budget from a sum into a max and avoids duplicating weights, at the cost of serialising the two phases. On one 8-GPU node with a 7B model, colocate; with a 70B model and a large rollout pool, disaggregate and async, then pay off the resulting staleness with importance-sampling correction. Follow-up: “what breaks in the async case?” -> staleness, engine/trainer log-prob mismatch, and sampling truncation changing the action space; the answers are ratio truncation, rollout_is correction, and keeping the sampling mask.

“Why is Ray in every one of these frameworks?” Strong answer: because the resource shapes are heterogeneous, dynamic and stateful: GPUs in fixed gang-scheduled groups for collectives, long-lived inference engines that must survive across steps, CPU-side environments and verifiers, and roles that come and go by phase. Ray gives actors over GPUs, placement groups for gang scheduling, and a rendezvous for torch.distributed in one layer. Where that is not true — fixed topology, one model, torchrun — Ray is pure overhead. Common wrong answer: “Ray makes training faster.” Ray adds about 0.47 ms of overhead per task; it does not speed up a kernel. It buys placement, not throughput.

“What actually changed by 2026?” Strong answer: three things. GRPO and its descendants became the default for reasoning/RLVR post-training (DeepSeek-V3.2’s flagship section is titled “Scaling GRPO”), TRL dropped PPO outright, and the systems frontier moved from synchronous colocated RL to asynchronous or partially off-policy RL with importance-sampling correction. PPO with a critic is not dead — it persists where the reward is dense and a value estimate is worth fitting — and DPO is still the default cheap offline method. Common wrong answer: “PPO is dead.” TRL removing its trainer is a statement about one library’s scope, not about the algorithm.

Next: KV Cache — the rollout half of every RL step is really a decode loop, and the KV cache is the memory term that decides how many completions you can have in flight.