Serving: vLLM and PagedAttention

TL;DR — A serving engine’s throughput is set by how many sequences fit in memory, and the naive design pre-allocates a contiguous KV tensor of max_len per request, so measured utilisation was only 20.4-38.2% — the rest was internal fragmentation, external fragmentation, and reserved slots (Kwon et al. 2023). PagedAttention splits the cache into fixed 16-token blocks held in a pool and reached through a per-sequence block table, which removes external fragmentation, caps internal fragmentation at one block, and lets sequences share prefix blocks copy-on-write. Continuous batching then re-forms the batch every decoding iteration instead of every request, which keeps every slot busy: $N=\lambda W$ means throughput at fixed latency is exactly a concurrency budget. The measured effect is 2.3× over Orca on beam search and up to 55.2% memory saved by sharing.

1. Why this exists

Two independent failures made high-throughput LLM serving hard in 2023, and both are about allocation and scheduling, not about the model.

Failure one: contiguous allocation. A framework that requires each request’s KV cache to be one contiguous tensor must size it before the first token is generated, and the only bound available is the maximum sequence length — so a request that will produce 120 tokens reserves 2,048. The PagedAttention paper finds that only 20.4% to 38.2% of the KV cache memory is used to store real token states in the systems it measured. The waste is not slack that is eventually filled: the reservation is held for the request’s whole lifetime, so nobody else can use it. When 90% of answers are under 300 tokens and max_len is 8,192, you pay 27× the memory you need — and memory caps batch size.

Failure two: request-level batching. The classic rule is to accumulate $B$ requests, run them together, and return results when the batch finishes. But LLM outputs have wildly different lengths, so short requests finish early and hold their slots idle until the longest sequence terminates, while new arrivals wait for the entire batch rather than for one slot. Orca (OSDI 2022) states it in the abstract: “requests that have finished earlier than other requests in a batch cannot return to the client, while newly arrived requests have to wait until the current batch completely finishes.” Orca was the fix, and reported 36.9× the throughput of FasterTransformer on GPT-3 175B at equal latency.

The two compound: wasted memory lowers $B$, and low $B$ with static batching lowers utilisation further. The rest of this chapter is the two fixes — paged memory for the cache, iteration-level scheduling for the batch — and what they made possible.

Scope. The KV Cache chapter is about how many bytes a sequence needs. This chapter is about how those bytes are laid out and scheduled, and why that determines your throughput.

2. How it works

2.1 The three kinds of waste, and their exact size

The paper names three, and they behave differently:

• Reserved slots. Memory held for tokens that will be generated (positions $L_{used}..L_{max}$). Not lost, but unavailable to anyone else meanwhile, so its cost shows up as low batch size rather than as waste.
• Internal fragmentation. Memory in the reservation that never gets used because the request finishes early. If true lengths $L$ are drawn from a distribution on $[0,L_{max}]$, this wastes the fraction $f_{\text{internal}}=1-\mathbb{E}[L]/L_{max}$.
• External fragmentation. Gaps left between variable-size allocations by the allocator (the paper points at buddy allocation). None of it is ever usable for a token; it is pure loss.

If actual lengths were uniform on $[0,L_{max}]$, $\mathbb{E}[L]=L_{max}/2$ and half the reservation is dead. Real output lengths are far more skewed — lognormal-ish with a long right tail — so with $L_{max}$ set for the tail, $\mathbb{E}[L]/L_{max}$ lands under 0.4. That is exactly the paper’s measurement, so 20.4-38.2% utilisation is arithmetic, not an implementation bug: it is what a fixed-size reservation implies.

2.2 Paging: the block table as virtual memory

PagedAttention imports the operating system’s answer to the same problem. The KV cache is divided into blocks of $B_b$ tokens (16 by default). Blocks live in one global pool of fixed-size physical slots, and a sequence’s cache is a list of logical blocks mapped to physical blocks by a per-sequence block table:

flowchart LR
    L0["logical block 0: tokens 0-15"] --> T0["table entry 0 -> physical 7"]
    L1["logical block 1: tokens 16-23"] --> T1["table entry 1 -> physical 3"]
    T0 --> P7["physical block 7"]
    T1 --> P3["physical block 3"]
    P9["physical block 9 (another sequence)"]

• Blocks are the unit of allocation, like OS pages. All blocks are the same size, so there is no external fragmentation — any free block fits any request.
• The block table is the page table: a logical index $j$ to a physical block id, plus how many slots in the last block are filled. It is tiny, one integer per 16 tokens per sequence.
• On-demand allocation means a request holds only the blocks it currently needs, filled left to right; a new physical block is requested only when the previous one is full. Reserved memory, as a category, disappears.

Now the fragmentation arithmetic changes character. With paging the only waste is the unfilled tail of the last block, $f_{\text{paged}}(L)=(B_b-(L\mathrm{mod}{B}_b))/B_b$, which averages over $L$ to $(B_b-1)/2$ dead tokens per sequence — a constant, not a fraction:

$$f_{\text{paged}}\approx \frac{B_b-1}{2\mathbb{E}[L]}$$

At $B_b=16$ the expected waste is $(16-1)/2=7.5$ tokens, so a 500-token answer pays 1.5% against $\ge 60\%$ under a 2,048-token reservation. The waste is now a constant per sequence rather than a fraction of the maximum.

Why 16 and not 1? A one-token block would waste nothing and be much slower. The paper’s ablation states the trade: “If the block size is too small, vLLM may not fully utilize the GPU’s parallelism for reading and processing KV cache. If the block size is too large, internal fragmentation increases and the probability of sharing decreases.” Blocks 16-128 won on ShareGPT, but on the shorter Alpaca traces only 16 and 32 worked “since the sequences become shorter than the block sizes”. Hence 16.

Why the indirection is nearly free. “Extra pointer chase in the hot loop” sounds fatal, but the gather happens inside the attention kernel, which already reads the cache in tiles; the table is a few hundred bytes per sequence and stays in cache; and tokens within a block are contiguous, so the kernel still gets coalesced 128-byte transactions. The paper states the boundary condition: paging pays when the workload is memory-capacity-bound with dynamic allocation, and for static or compute-bound workloads “introducing the vLLM’s techniques may rather degrade the performance due to the extra overhead of memory indirection”.

Sharing, and copy-on-write. Because blocks are referenced rather than copied, two sequences with the same prefix can point at the same physical blocks; when one writes into a shared block, refcounting triggers a copy first — OS copy-on-write. This makes beam search and parallel sampling cheap: 37.6-55.2% memory saved on beam search with the Alpaca trace and 44.3-66.3% on ShareGPT, against 6.1-9.8% and 16.2-30.5% for parallel sampling, which shares only the prompt.

2.3 Continuous batching, and the law that governs it

The second fix replaces request-level batching with iteration-level (or continuous) batching, Orca’s core contribution: the scheduler runs one decoding iteration of the whole batch, then re-forms the batch. Finished sequences are retired and their blocks freed; queued requests are admitted and get their prefill scheduled. A new request waits for one iteration, not for a whole batch.

The right way to reason about this is Little’s law: $N=\lambda W$, where $N$ is the average number of requests in the system (here, sequences resident in the engine, including preempted ones), $\lambda$ is throughput in completed requests per second, and $W$ is the mean time a request spends in the system in seconds.

It is an accounting identity, not a model: for any stable system the average number in the system equals the arrival rate times the average residence time, with no assumption about the length distribution. Rewritten,

$$\lambda =\frac{N}{W}$$

which should govern your capacity planning: throughput is concurrency divided by latency, and you cannot move one without the other. Since $N$ is capped by KV memory (and by max_num_seqs), raising $\lambda$ at fixed hardware means raising $N$, which raises $W$ — more load per step, longer TPOT. There is no free throughput. Now compare the two batching disciplines under that law.

Static batching does not hold $N$ constant; it lets $N$ decay. A batch of $B$ starts full and finishes at $T_{max}={\max }_i{T}_i$, the longest member. If the $T_i$ are i.i.d. exponential with mean $\bar{T}$, then $\mathbb{E}[T_{max}]=\bar{T}{H}_B$ with $H_B={\sum }_{i=1}^{B}1/i$, so slot-time utilisation is mean over max:

$$U_{\text{static}}=\frac{\bar{T}}{\bar{T}{H}_B}=\frac{1}{H_B}$$

(example) $B=8\Rightarrow H_8=2.72\Rightarrow U=37\%$; $B=32\Rightarrow H_{32}=4.06\Rightarrow U=25\%$. Utilisation gets worse as the batch grows, because the maximum of more samples runs further into the tail. That is the quantitative content of “the GPU idles waiting for the slowest request”.

Continuous batching refills each freed slot on the next iteration, so while the queue is non-empty every one of the $N_{max}$ slots is occupied and $U\to 1$. The same $N_{max}$ now buys roughly $H_B$ times the throughput at the same latency — the mechanism behind Orca’s 36.9× headline, though that number also includes kernel and distributed-execution work, so treat it as an upper bound on the combination rather than on scheduling alone.

The token-level version of the same argument is the ceiling derived in KV Cache §2.5: aggregate tokens/s saturates at $\text{BW}/(sk)$, which falls as $1/s$ and no amount of batching or scheduling can lift. That is why long-context serving is a memory problem, and why the frontier moved to GQA, MLA, and fp8 KV.

2.4 Chunked prefill: keeping the batch homogeneous

A decode step is a batch of $N$ single-token forwards; a prefill is a different shape — one sequence with thousands of tokens. If the scheduler runs a long prefill as its own step, every in-flight decode stalls for its duration. For a 70B, a 10k-token prefill is roughly $2Pn=2\times 70.6\times 10^9\times 10^4\approx 1.4\times 10^{15}$ FLOPs; at a realistic 30% of an H100’s 989 TFLOP/s that is ~4.7 s of stalled inter-token latency for every other user.

Chunked prefill (Sarathi-Serve, arXiv:2403.02310) splits the prefill into chunks of at most max_num_batched_tokens and interleaves them with decode steps, so the GPU never runs one giant irregular kernel. The trade is explicit: the long prompt’s own TTFT gets worse (spread over several iterations) while everyone else’s p99 ITL gets much better. Good for a chat service, bad for a batch summariser.

2.5 Preemption: recompute or swap

Paging removes waste, not the capacity limit. When the pool is exhausted and a running sequence needs a new block, the scheduler must preempt someone: recomputation drops the victim’s blocks and re-runs its prompt prefill when it resumes (cost: prefill FLOPs), while swapping copies the blocks to CPU RAM over PCIe (cost: transfer).

The paper’s §7.3 is unusually decisive: recomputation overhead “is never higher than 20% of swapping’s latency”, because swapping a large paged cache over PCIe is bandwidth-limited while recomputation is compute on an otherwise memory-bound engine. Swapping wins only for large blocks, where transfers amortise; for block sizes 16 to 64 the two are comparable. vLLM defaults to recomputation.

The scheduler also tries not to get there: scheduler_reserve_full_isl (default True) checks that a new request’s entire input fits in the KV cache before admitting it, rather than only its first chunk. Without that check, a long prompt admitted on a small first chunk can consume blocks as it prefills and force preemptions of everything else.

2.6 Reusing prefixes across requests

A system prompt, a few-shot block, a document queried repeatedly, or a chat history are all identical prefixes. Because the cache is not content-addressed (KV Cache §2.1), reuse is only possible at exact prefix granularity — exactly what a block table provides: hash each block’s tokens, look the hash up in a global table, and point at the existing block instead of computing it.

The elegant version is RadixAttention in SGLang (arXiv:2312.07104), which keeps the hashes in a radix tree so a lookup finds the longest matching prefix rather than a fixed-length one. Its measured effect is “up to 6.4× higher throughput compared to state-of-the-art inference systems” across multi-turn chat, RAG pipelines, and JSON decoding — all high-overlap workloads. With no shared prefix there is nothing to reuse and the hash bookkeeping is pure cost.

2.7 Speculative decoding

Decode is memory-bound, so the GPU has spare FLOPs while it waits on bandwidth. Speculative decoding spends them: a cheap draft proposes $\gamma$ tokens and the target verifies all of them in one forward pass, because verifying is just a batched forward over the candidate continuation.

The acceptance rule is what makes it lossless. Proposal $x$ comes from the draft distribution $q(x)$ but should come from the target’s $p(x)$; accept with probability $\min (1,p(x)/q(x))$ and, on rejection, sample from the normalized residual $\max (0,p(x)-q(x))$. This is speculative sampling (Leviathan et al., Chen et al.) and it produces exactly the target model’s distribution — no quality loss, unlike every other decode-acceleration trick in this book.

The payoff follows from acceptance. Let $\alpha$ be the per-token acceptance probability and $\gamma$ the number of drafts. The expected number of tokens per target forward is one (the target’s own token) plus the expected number of accepted drafts, a truncated geometric sum:

$$E[\text{tokens}]=1+\alpha +{\alpha }^2+\cdots +{\alpha }^{\gamma }=\frac{1-{\alpha }^{\gamma +1}}{1-\alpha }$$

Check the ends: $\gamma =0\Rightarrow E=1$ and $\alpha =0\Rightarrow E=1$. With $\alpha =0.8,\gamma =4$: $E=(1-0.32768)/0.2=3.36$ tokens per target forward. Real speedups are lower (typically 1.5–2.5×) because the verification pass costs more than a plain decode step and the draft model is not free.

[as of 2026-10] vLLM 0.31 exposes this through --speculative-config with methods draft_model, eagle3, mtp, ngram, suffix, mlp, and pard; its docs characterise the model-based methods as the best latency reduction and n-gram/suffix as “modest speedups without increasing workload during peak traffic”.

3. Implementation

vllm==0.31.0 [as of 2026-10], the release current at the time of writing.

3.1 Offline: one process, every knob visible

# pip install vllm==0.31.0
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="bfloat16",     # weight/activation precision; 2 bytes per KV element
    max_model_len=8192,   # the L_max that used to cause the fragmentation
    gpu_memory_utilization=0.90,   # default 0.92; a hard budget, never 1.0
    max_num_seqs=64,      # N cap; effective default 1024 on H100/H200, 256 otherwise
    max_num_batched_tokens=8192,   # tokens per scheduler step = one prefill chunk bound
    enable_chunked_prefill=True,   # default on for decoder-only models
    enable_prefix_caching=True,    # default on; hashes blocks for cross-request reuse
    block_size=16,        # default 16; tokens per KV block
    kv_cache_dtype="auto",  # "fp8" halves the cache and the bytes read per step
)

params = SamplingParams(
    temperature=0.7,   # 1.0 = raw distribution, 0.0 = greedy
    top_p=0.9,         # nucleus cutoff; 1.0 = off
    max_tokens=512,    # DEFAULT IS 16. Always set it explicitly.
    n=1,               # >1 = parallel sampling; prompt blocks are shared
    seed=0,            # ties a run to a reproducible token stream
)

outs = llm.generate(["Summarise paged attention in one paragraph."], params)
print(outs[0].outputs[0].text)

One version trap worth knowing: older configs pass swap_space, which no longer exists. The replacement is kv_offloading_size (GiB) plus kv_offloading_backend, and the old argument is rejected at startup rather than silently ignored. That is the kind of detail to check against the release’s own source, not a blog post.

3.2 Online: the OpenAI-compatible server

# Serves /v1/chat/completions and /v1/completions.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 4096 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --served-model-name llama31-8b

# pip install openai
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="llama31-8b",
    messages=[{"role": "user", "content": "Explain continuous batching."}],
    max_tokens=512,
    extra_body={"ignore_eos": True},   # vLLM extensions go here
)
print(resp.choices[0].message.content)

On startup, read two things. First the KV capacity: GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X.XXx (v0.31.0 source). Second the prefix-cache hit rate, exposed as a Prometheus metric; if it is near zero, --enable-prefix-caching is buying you nothing.

3.3 Turning on speculative decoding

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --speculative-config '{
    "method": "draft_model",
    "model": "meta-llama/Llama-3.2-1B-Instruct",
    "num_speculative_tokens": 5
  }'

num_speculative_tokens is $\gamma$. It is a sample-efficiency knob: larger $\gamma$ raises the ceiling $1/(1-\alpha )$ but adds draft cost and lowers $\alpha$ if the draft is weak. Tune it against measured acceptance length, not against the theoretical curve. The same JSON is accepted from Python as LLM(..., speculative_config={...}).

4. Numbers and trade-offs

4.1 What the paper actually measured

Two things to notice. The sharing numbers are much larger for beam search than for parallel sampling, because beams share intermediate hypotheses while independent samples share only the prompt. And the abstract’s “2-4×” is smaller than the text’s “up to 22×” request-rate figure: those are different measurements (throughput at equal latency versus sustainable request rate), and quoting the bigger one without its condition is the usual sleight of hand.

4.2 Throughput versus latency, and the only honest metric

Batching trades latency for throughput, so a single number is meaningless. Model the decode step as moving weights + N caches bytes:

$$\text{step time }\tau (N)\approx \frac{Pp+Nsk}{\text{BW}},\text{tokens/s}=\frac{N}{\tau (N)}$$

Throughput rises linearly in $N$ while the weight read dominates and then saturates, while per-request TPOT grows roughly linearly in $N$ once the KV term dominates, because every extra sequence adds $sk$ bytes to every step for everyone. The knee of that curve is where you operate, and it moves with context length. That is why goodput — output tokens/s that still satisfy the TTFT and TPOT constraints — is the metric to report; bare throughput is trivially inflated by admitting more requests and letting latency blow up.

5. Pitfalls

gpu_memory_utilization near 1.0. Symptom: CUDA OOM during startup or CUDA-graph capture although the weights loaded. vLLM gives the remaining budget to the KV pool; at 0.99 nothing is left for the transient activation peak, the graph capture, or a co-located process. Use 0.85-0.92, and roughly half each if two engines share a GPU.

Benchmarking at concurrency 1 and concluding batching “doesn’t help”. Symptom: a flat throughput-vs-concurrency plot. At $N=1$ you are measuring the weight-read bandwidth wall (see KV Cache §2.4). Sweep at least three concurrency levels.

Preemption thrash. Symptom: throughput falls as you raise concurrency, with Preemption counters climbing and ITL spikes. Once $N$ exceeds what the KV pool holds at the current context length, every step preempts and re-prefills someone. Lower max_num_seqs or max_model_len; do not raise memory.

Block size too large for short outputs. Symptom: prefix-cache hit rate is fine but throughput is worse than with a smaller block, and logs show many partially filled blocks. With 200-token answers and 128-token blocks, most of a block is dead per sequence — the paper measured exactly this on Alpaca.

Turning on speculative decoding at high QPS. Symptom: throughput drops under load even though single-stream latency improved. Speculation spends compute to save bandwidth — the right trade exactly when you are bandwidth-bound at low concurrency; at high concurrency the batch already consumes the compute budget and the draft plus verification kernels are pure overhead. Always report acceptance length alongside the claim.

6. Using it in a project

Reach for a paged, continuously-batched engine when: you self-host, you have concurrent traffic (the crossover is roughly “more than a handful of in-flight requests”), outputs vary in length, and the KV cache is a meaningful share of GPU memory. That is the standard chat, agent, and RAG-serving case.

Do not reach for it when:

• You have one request at a time. A desktop assistant or an internal tool at 1-2 QPS gets nothing from paging or continuous batching; the engine is bandwidth-bound on the weight read either way, and a plain transformers loop or a small local runtime is simpler.
• You have an offline batch job. Use vLLM’s offline LLM class — you want the throughput, not the server. Standing up vllm serve, a load balancer, and a health check for a nightly summarisation job is three moving parts too many.
• Your prompts are short and uniform. If every request is 200 tokens in and 200 out, fragmentation was never the problem and prefix overlap is nil; the advanced features are inert.
• Your bottleneck is retrieval or output parsing. Paged attention changes the generation step. If p95 is dominated by an embedding call, a reranker, or JSON repair, you are optimising the fast part. See The RAG Pipeline.
• A managed API is cheaper. At low volume per-token pricing beats an idle GPU; the self-hosting crossover is about utilisation, not capability.
• The model fits with room to spare and traffic is bursty. Paging and prefix caching add failure modes to solve a capacity problem you do not have.

Decision order: (1) measure goodput at three concurrency levels and find the knee; (2) cap max_model_len at the product’s need; (3) enable prefix caching if prefixes repeat and verify the hit rate; (4) enable chunked prefill if p99 ITL matters and long prompts are mixed in; (5) only then consider fp8 KV and speculative decoding, after measuring their accuracy and acceptance costs.

7. How you would prove it worked

Metric. Goodput at a stated SLO, reported as a triple: TTFT p50/p99, TPOT p50/p99, and output tokens/s. Add the prefix-cache hit rate and the preemption count as explanatory variables — they tell you why the number moved.

Baseline. Same model, same max_model_len, same arrival trace and concurrency, change reverted. A serving change without a load generator is an anecdote.

Ablation. Turn one thing off at a time; four experiments, one per mechanism in §2:

1. Block/paging. If your old stack is available, compare against a contiguous allocator at the same model and trace. The difference should show up as resident-sequence count, matching the paper’s 20.4-38.2% story.
2. Continuous vs static batching. Fix concurrency and compare against a request-level batcher on a length-varied trace. The predicted gain is roughly $H_B$ (example: ~2.7× at $B=8$, ~4× at $B=32$); a much smaller gain means your lengths are nearly uniform, not that the technique failed.
3. Prefix caching on/off on real traffic, reading the hit-rate metric. A null result plus a ~0% hit rate is a data property, not an engine property.
4. Speculative decoding at low and high concurrency, reporting acceptance length. Expect the gain to invert between the two regimes; that inversion is the result. Reporting template (example — replace with your own numbers).

“On 1×H100, Llama-3.1-8B, 8k cap, a mixed-length trace: at max_num_seqs=64 we sustained 1,850 output tok/s (example) at p99 TPOT 45 ms, against 1,200 tok/s (example) at max_num_seqs=32 and the same p99. The prefix-cache hit rate was 71% (example) on the chat trace and 4% (example) on the unique-document trace, which is why it helped the first and not the second. Speculative decoding (EAGLE-3, num_speculative_tokens=4) cut p50 TPOT 38% (example) at concurrency 1 and cost 9% throughput at concurrency 64, so we enabled it only on the interactive endpoint.”

8. Interview drill

“Why was naive KV-cache management wasting 60-80% of the cache?” Strong answer: both mechanisms come from one contiguous tensor per request sized for the maximum length. Internal fragmentation: the reservation is sized for max_len but the request finishes earlier, so the tail is dead — with lengths uniform on $[0,L_{max}]$ that alone is a 50% loss. External fragmentation: variable-size allocations leave unusable gaps. Combined, the paper measures 20.4-38.2% utilisation. Common wrong answer: “the cache is too big” — the waste is in tokens that never exist.

“What does continuous batching buy you over static batching?” Strong answer: it holds $N$ constant instead of letting it decay. Under static request-level batching the batch lives until its longest member finishes, so slot utilisation is mean over max; for i.i.d. exponential lengths that is $1/H_B$ — about 37% at $B=8$ and 25% at $B=32$, i.e. worse with bigger batches. Iteration-level batching retires and admits per step, so every slot stays busy and Little’s law applies cleanly: $\lambda =N/W$. Common wrong answer: “it batches requests that arrive together, so there’s less padding.” That is sequence padding; the win is constant occupancy.

“How does a block table not destroy performance?” Strong answer: the gather is inside the attention kernel, which already reads the cache in tiles; the table is a few hundred bytes per sequence and stays hot; and tokens within a block are contiguous, so transactions stay coalesced. Follow-up: “when is it a loss?” → compute-bound or static-shape workloads, where you pay indirection for flexibility you do not need.

“How do you choose block size?” Strong answer: it trades kernel efficiency against fragmentation and sharing granularity. Too small and the kernel cannot parallelise its reads; too large and each sequence wastes more of its last block and prefix sharing only triggers at block boundaries, so hit probability falls. The paper found 16-128 best on ShareGPT but only 16-32 on the shorter Alpaca traces, and ships 16. Common wrong answer: “smaller is always better” — that ignores the kernel and the sharing granularity.

“Recompute or swap on preemption?” Strong answer: recompute, at the default block sizes. The paper measures recomputation overhead as never more than 20% of swap latency, because swap is PCIe-bandwidth-bound while recompute is FLOPs on an engine that is bandwidth-bound anyway. Swap becomes competitive for large blocks. Common wrong answer: “swap, because the cache is precious” — it is precious precisely because moving it is expensive.

“Throughput or goodput?” Strong answer: goodput — tokens/s subject to a TTFT and TPOT SLO. Bare throughput can be inflated by admitting more requests and letting latency grow, so it is not comparable across configurations. By Little’s law every throughput claim implicitly claims a latency.

Next: Ray — a single paged engine is one process on one node; Ray is the layer that runs many of them, and the RL loops that keep feeding them.