KV Cache

TL;DR — Autoregressive decoding would recompute the keys and values of the whole prefix on every step, making generation $O(n^2)$ per sequence. Caching them makes it $O(n)$ time but $O(n)$ memory: $2L{n}_{kv}{d}_{h}sBp$ bytes, dominated by $L$ and the sequence length $s$. GQA reduces only $n_{kv}$, in exact proportion ($64/8=8\times$ for Llama-3-70B), and MLA compresses the cache into a latent instead. Decode re-reads every weight for every token, so its arithmetic intensity is roughly $2B/p$ FLOP/byte against an H100 ridge point of 295 — it is memory-bandwidth-bound, while prefill, which serves $n$ tokens per weight read, is compute-bound. At 128k context, four concurrent 70B-class requests hold more KV bytes than the model’s own weights.

1. Why this exists

Start with the naive decode loop, because the cache is only justified by what it removes. To generate token $n+1$ you need its query to attend over the keys and values of tokens $\mathrm{1..}n$. If you do not cache anything, you recompute $K$ and $V$ for the entire prefix at every step. For a sequence of length $n$ that is

$$\text{work}=\sum _{t=1}^{n}t\approx \frac{n^2}{2}\text{ key/value projections per sequence}$$

per layer, per head. That is the quadratic term Attention warned about, and it is what makes un-cached generation unusable at long context: generating 8,192 tokens costs about $8192^2/2\approx 3.4\times 10^7$ projection-steps instead of $8192$.

Caching fixes the compute. It creates the memory problem that the rest of this chapter is about, and the two are the same coin: you trade repeated arithmetic for retained state. The retained state is the KV cache, and at manageable batch sizes it is the largest dynamic allocation on the GPU. The PagedAttention paper makes this concrete for a 13B model on a 40 GB A100: about 65% of memory is weights, close to 30% is KV cache, and the rest is activations (Figure 1, left).

A second failure appears once you batch. Decode reads a fixed quantity of weights per step regardless of how many sequences are in the batch, so batch size is free throughput — until the KV cache for those sequences no longer fits. Memory capacity, not FLOPs, then sets the batch size, and batch size sets the throughput. That is the sentence to carry out of this chapter.

Where this leaves you. Attention §4 already gives the byte formula and a 7B example. This chapter justifies every factor in that formula, derives the arithmetic intensity behind prefill and decode, computes a real 70B-class model, and prices each mitigation (GQA, MLA, KV quantization, prefix caching, eviction) against the factor it moves.

2. How it works

2.1 What exactly gets cached

For each layer, each token produces $q$, $k$, and $v$; only $k$ and $v$ are cached.

• $k$ and $v$ for position $t$ are reused by every future token. Token $n+1$, $n+2$, $\dots$ all attend back to position $t$, so its key and value must survive.
• $q$ for position $t$ is used exactly once, by token $t$ itself. Nothing later reads it. Caching it would be pure waste, which is why the factor is 2 and not 3.

This asymmetry is why the factor is 2 and not 3, and it explains what does not shrink the cache: sparse attention, a different softmax, or more query heads all leave the number of distinct $(k,v)$ vectors unchanged.

One load-bearing subtlety: the cache is not content-addressed. The cached key for the token "the" depends on every token before it, so the same string at two positions has two different cached vectors. Prefix caching can therefore only ever reuse an identical prefix, never a repeated phrase.

flowchart LR
    subgraph step_t["decode step t"]
        X["new token embedding"] --> Q["q_t"]
        X --> K["k_t"]
        X --> V["v_t"]
    end
    K --> CACHE[("KV cache: k_1..k_t, v_1..v_t")]
    V --> CACHE
    CACHE --> ATT["attention: q_t against ALL cached k"]
    Q --> ATT
    ATT --> OUT["weighted sum over ALL cached v"]

2.2 The byte formula, symbol by symbol

$$M_{KV}=2\cdot L\cdot n_{kv}\cdot d_h\cdot s\cdot B\cdot p$$

$M_{KV}$ is total KV-cache bytes across all resident sequences: a count of stored scalars times a bytes-per-scalar. Every factor is a place a different mitigation acts.

• $2$ — keys and values. There is one $k$ tensor and one $v$ tensor of identical shape per layer, per token. Remove it and you have cached half of what attention needs, which is not an optimisation.
• $L$ — number of transformer layers. Each layer has its own attention parameters and therefore its own $k$ and $v$ for every token; nothing is shared across depth. This is why the 126-layer Llama-3-405B has a cache 1.575× that of an 80-layer model of the same width. You cannot reduce $L$ without changing the architecture.
• $n_{kv}$ — number of key/value heads per layer. This is the only factor GQA and MQA move, which is why they are the highest-leverage serving decision available to you. In Multi-Head Attention $n_{kv}=n_{heads}$; in GQA the $n_{heads}$ query heads are partitioned into $n_{kv}$ groups sharing one K/V head each; in MQA $n_{kv}=1$. The cache shrinks by exactly $n_{heads}/n_{kv}$ because the shape of the stored tensor changes — a GQA model genuinely never materialises the per-query-head keys. For Llama-3-70B, $64/8=8\times$.
• $d_h$ — per-head width, usually 128. Fixed by the checkpoint, not tuned at serving time. It is the axis on which MLA’s compressed latent is best compared: 576 elements against a full $n_{kv}{d}_h=8\times 128=1024$.
• $s$ — sequence length in tokens, prompt plus generated. Unlike the other factors, $s$ is partly yours: it is the context you admit. The cache is linear in $s$, which is why an enormous context window is a memory story before it is a quality story.
• $B$ — number of concurrently resident sequences. Each additional resident sequence costs one full $2L{n}_{kv}{d}_{h}sp$, and nothing is shared unless prefixes are identical.
• $p$ — bytes per element: 2 for fp16/bf16, 1 for fp8/int8, 0.5 for 4-bit. fp8 halves $M_{KV}$ and halves the bytes read per decode step.

Two consequences fall out, and they are the usual interview traps: batching multiplies $M_{KV}$ linearly (there is amortisation in the weights, none in the cache), and only $n_{kv}$, $p$, $s$, and $B$ are under your control — $L$, $d_h$, and the factor 2 are properties of the model.

2.3 Worked example: Llama-3.1-70B

Geometry from the Llama 3 paper (Table 3, arXiv:2407.21783): $L=80$, $n_{heads}=64$, $n_{kv}=8$, model width 8192 so $d_h=8192/64=128$. Llama-3.1 keeps that geometry and extends the context to 128k with RoPE scaling.

Per token, in bf16 ($p=2$):

$$2\times 80\times 8\times 128\times 2\text{B}=327,680\text{B}\approx 0.328\text{MB/token}$$

That single number is the one to memorise for this model family. It scales into everything else:

($s\times 0.32768$ MB; decimal GB.)

The model’s own weights in bf16 are $70.6\times 10^9\times 2\approx 141$ GB. Read the table against that number:

• At the native 8k context, 52 concurrent sequences cost the same as the weights (52 × 2.68 = 139.4 GB); at 64 concurrency the cache is larger. Serving a 70B at 8k with real concurrency makes the cache co-equal with the weights.
• At 128k context, four sequences cost more than the weights, and batch 8 is 2.4× the weights: context length, not batch size, is what kills you here.
• With plain MHA instead of GQA ($n_{kv}=64$) the per-token figure becomes 2.62 MB, so one 128k sequence needs 343.6 GB — more HBM than a four-GPU node for a single conversation. GQA is the difference between fitting and not.

2.4 Why decode is memory-bandwidth-bound and prefill is compute-bound

The roofline model says a kernel is limited by whichever is smaller: the compute peak (FLOP/s), or bandwidth times arithmetic intensity (FLOP/byte). The crossover is the ridge point

$$I^{\ast }=\frac{\text{peak FLOP/s}}{\text{bandwidth}}$$

(A100 and H100 figures as published by NVIDIA; H200 shares the H100 compute die and spends its gains on HBM3e, so its ridge point moves down — the 989 TFLOP/s in that row is inferred from H100, not quoted from the H200 page.)

Now compute the intensity of both regimes for our 70B at $s=8192$. Define:

• $P\approx 70.6\times 10^9$ — non-embedding parameters. Every one of them is read once per forward pass.
• $\text{FLOPs per token}=2P+4L{d}_{model}n$. The first term is the projection and FFN work: one multiply-add (2 FLOPs) per parameter per token. The second is attention: $2L{d}_{model}n$ for $Q{K}^{\top }$ and the same again for the value mixing. At $s=8192$ this second term is $2.15\times 10^{10}$, about 15% of the first.
• Bytes moved per decode step $=Pp+B\cdot s\cdot k$, where $k=2L{n}_{kv}{d}_{h}p=327,680$ B is the per-token-per-sequence cache cost. The weights are re-read entirely for each token; the KV term is what grows with batch.

For prefill at batch 1 with $n$ tokens, the weights are read once and $n$ tokens are processed:

$$I_{\text{prefill}}\approx \frac{n(2P+4L{d}_{model}n)}{Pp+nk}=\frac{8192\times 1.627\times 10^{11}}{1.412\times 10^{11}+2.68\times 10^9}\approx 9000\text{FLOP/byte}$$

That is 31× past the H100 ridge point: prefill is firmly compute-bound, with a floor of $1.33\times 10^{15}/989\times 10^{12}\approx 1.35$ s for 8,192 tokens, about 0.16 ms of compute per token.

For decode at batch 1, one token is processed per full weight read:

$$I_{\text{decode}}\approx \frac{2P+4L{d}_{model}s}{Pp+sk}=\frac{1.627\times 10^{11}}{1.412\times 10^{11}+2.68\times 10^9}\approx 1.13\text{FLOP/byte}$$

That is 260× below the ridge point. The floor is the memory time, $1.44\times 10^{11}/3.35\times 10^{12}\approx 43$ ms per token, or ~23 tokens/s — the same ceiling Quantization quotes for the 70B’s weight read, because it is the same bandwidth wall. Prefill costs ~0.16 ms of compute per token; decode costs ~43 ms. That 260× gap is the entire reason TTFT and TPOT are separate metrics.

flowchart LR
    A["weights: 141 GB read once per decode step"] --> C["decode step"]
    B["one token of FLOPs: 163 GFLOP"] --> C
    C --> D["intensity 1.13 FLOP/byte"]
    D --> E["H100 ridge 295"] --> F["memory-bound by ~260x"]

2.5 Batching helps, but only up to a ceiling

Batching amortises the weight read but not the KV read, so the asymptotic intensity as $B\to \infty$ is $I_{\infty }(s)=f(s)/(sk)$ with $f(s)=2P+4L{d}_{model}s$. Setting $f(s)=I^{\ast }sk$ with $I^{\ast }=295$ and $k=327,680$ gives $s^{\ast }\approx 1,500$ tokens. Below that, enough batching makes 70B fp16 decode genuinely compute-bound on an H100. Above it, no batch size reaches the ridge: at 8k the asymptotic intensity is only 60 FLOP/byte, 4.9× short, because the KV read grows linearly with $B$ exactly as the FLOPs do. Long-context decode is memory-bound no matter how you schedule it, which is why every mitigation that matters there is a memory mitigation.

The same algebra gives the token-rate ceiling. Per decode step the batch advances $B$ tokens while moving $Pp+Bsk$ bytes:

$$\text{tokens/s}=\frac{B\cdot \text{BW}}{Pp+Bsk}\stackrel{B\to \infty }{\to }\frac{\text{BW}}{sk}$$

For the 70B at 8k on an H100 that ceiling is $3.35\times 10^{12}/2.68\times 10^9\approx 1,250$ tokens/s, regardless of batch size. You reach half of it when $Pp=Bsk$, i.e. $B=52.6$ — the batch at which the cache equals the weights. A consistency check, not a fluke.

2.6 The mitigations and exactly which factor they move

MLA is worth one number. DeepSeek-V3 uses 61 layers with kv_lora_rank = 512 and qk_rope_head_dim = 64 (config), so it caches $512+64=576$ elements per token per layer instead of $n_{kv}{d}_h=8\times 128=1024$, giving $61\times 576\times 2=70,272$ B/token. That is 4.7× smaller than the 70B GQA figure and 37× smaller than 70B MHA. The DeepSeek-V2 paper reports the aggregate effect directly: MLA “reduces the KV cache by 93.3%” and “boosts the maximum generation throughput to 5.76 times” versus DeepSeek 67B (arXiv:2405.04434).

3. Implementation

Everything here is vllm==0.31.0 [as of 2026-10]. The point of the snippet is that the memory knobs are the model config knobs, so the numbers you compute in §2 are the numbers you set.

3.1 Computing the budget before you launch

# Pure arithmetic, no GPU needed. Use it to size a deployment.
def kv_bytes_per_token(n_layers, n_kv_heads, d_head, bytes_per_element):
    return 2 * n_layers * n_kv_heads * d_head * bytes_per_element   # formula from 2.2

def kv_total_gb(n_layers, n_kv_heads, d_head, bytes_per_element, seq_len, batch):
    return kv_bytes_per_token(n_layers, n_kv_heads, d_head, bytes_per_element) \
        * seq_len * batch / 1e9                       # decimal GB

LLAMA31_70B = dict(n_layers=80, n_kv_heads=8, d_head=128)
LLAMA31_8B  = dict(n_layers=32, n_kv_heads=8, d_head=128)

print(kv_bytes_per_token(**LLAMA31_70B, bytes_per_element=2))   # 327680
print(kv_total_gb(**LLAMA31_70B, bytes_per_element=2, seq_len=131072, batch=4))  # 171.8
print(kv_total_gb(**LLAMA31_8B,  bytes_per_element=2, seq_len=8192,  batch=16))  # 17.2
print(kv_total_gb(**LLAMA31_8B,  bytes_per_element=1, seq_len=8192,  batch=16))  #  8.6

Two calibration points: Llama-3-8B is 128 KiB per token, not the 512 KB the Part 04 README quotes — that figure is LLaMA-2-7B with 32 KV heads and no GQA. And fp8 halves the second number exactly, since $p$ is linear.

3.2 A minimal engine configured around the cache

# pip install vllm==0.31.0        # [as of 2026-10]
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    dtype="bfloat16",        # p = 2. bf16 is the default for bf16 checkpoints.
    max_model_len=8192,      # s cap. This is the single biggest lever on M_KV.
    gpu_memory_utilization=0.90,  # fraction of HBM vLLM may claim; default 0.92.
    kv_cache_dtype="auto",   # "auto" = bf16; "fp8" halves p. See 3.3.
    max_num_seqs=64,         # B cap: never more than 64 sequences resident.
    enable_prefix_caching=True,   # reuse KV for identical prefixes.
)

params = SamplingParams(
    temperature=0.7,   # 0.0 = greedy. Default 1.0.
    top_p=0.9,         # nucleus sampling. Default 1.0 (off).
    max_tokens=512,    # DEFAULT IS 16. Always set this.
    seed=0,            # makes a cache-on vs cache-off comparison reproducible.
)

for out in llm.generate(["Explain paged attention in two sentences."], params):
    print(out.outputs[0].text)

What each knob does to the numbers in §2:

• max_model_len sets the ceiling on $s$: the number in the KV arithmetic and the concurrency vLLM will admit. Halving it doubles resident sequences.
• gpu_memory_utilization decides how much HBM vLLM may claim in total (weights + KV + activations + CUDA graphs). It does not set the cache size; after profiling the weights and a worst-case activation footprint, vLLM gives everything left to the KV cache, so this one number sets $B_{\max }$. Range in practice: 0.85–0.92. See §5 for why 0.99 is a bug.
• max_num_seqs sets an explicit $B$ cap: if the cache would allow 300 sequences, this can still hold you to 64 — good for latency SLOs, wasteful for throughput.
• kv_cache_dtype="fp8" halves $p$ and therefore $M_{KV}$, at the accuracy cost in §3.3.
• enable_prefix_caching makes $s$ effective rather than nominal: a shared 3,000-token system prompt is paid for once.
• max_tokens=512: the SamplingParams default is 16, so forgetting it silently truncates every response.

After startup, vLLM logs the resolved capacity as GPU KV cache size: N tokens, Maximum concurrency for M tokens per request: X.XXx (v0.31.0 source). That concurrency is $B_{\max }$ at the full max_model_len: §2’s table, measured rather than predicted. If it is below what you need, the fixes in order of leverage are lower max_model_len, prefix caching, kv_cache_dtype="fp8", then a GQA/MLA checkpoint. Raising gpu_memory_utilization is the one fix that does not exist, because it is already handing over everything it can.

3.3 KV-cache quantization, and when it pays

vllm serve <model> --kv-cache-dtype fp8 stores the cache in fp8 e4m3 and runs the attention matmuls in fp8 on supported backends. The vLLM team’s validation [as of 2026-10] reports, for Llama-3.1-8B on an H100, that the decode inter-token-latency slope falls to 54% of bf16 (4.37e-05 → 2.37e-05 ms per token of context) with an intercept penalty of only 0.14 ms, so FP8 wins beyond a break-even of about 7k tokens; that under load (concurrency 8, ~20k-token inputs) output throughput rises 14.9% and median ITL falls 14.8%; and that accuracy with uncalibrated scales costs 1–2 points on reasoning benchmarks while recovering 97–98% of baseline AUC on MRCR@128k.

The same report is equally clear about when it loses: contexts below ~7k tokens (the constant overhead is not amortised), models with head_dim = 256 where prefill TTFT rises ~1.6×, and models with many small sliding-window layers, where skipping those layers (--kv-cache-dtype-skip-layers sliding_window) is better than quantizing them. See The State of FP8 KV-Cache in vLLM.

4. Numbers and trade-offs

Axis discipline. Name the axis a decision moves. GQA and MLA move memory capacity and, through it, concurrency. Quantization moves bandwidth and capacity. Batching moves throughput at the cost of latency and capacity. Context length moves quality at the cost of both.

4.1 Per-token cost across the Llama-3 family

The last row is why “a 7B costs 0.5 MB/token” was true in 2023 and is 4× too pessimistic for a modern 8B: GQA happened in between.

4.2 Where the cache overtakes the weights

Weights in bf16 are $2P$ bytes and the cache is $sBk$ bytes, so they are equal at

$$B_{\text{equal}}=\frac{2P}{sk}$$

The ceiling column is $\text{BW}/(sk)$ from §2.5 at H100 bandwidth, and the last column is where throughput reaches half of it — algebraically the same batch at which the cache equals the weights. Both are ceilings, not forecasts: they assume every KV byte is read at full bandwidth and ignore the weight read, and achieved throughput is typically 30–60% of them (example). The value is the shape: per-GPU throughput falls as $1/s$, and fp8 doubles both the ceiling and the crossover batch. For an 8B, 15 concurrent 8k conversations already cost as much cache as weights; for a 70B the crossover is 52 sequences at 8k but only 3.3 at 128k. Long context is where the cache becomes the model.

5. Pitfalls

Sizing memory from the weights and forgetting the cache. Symptom: the model loads fine, then vLLM logs a Maximum concurrency far below your expected QPS, or you OOM as traffic ramps. The weights are the static allocation; the cache grows.

Setting gpu_memory_utilization too high. Symptom: a CUDA OOM during engine startup or CUDA-graph capture, even though the model itself loads. vLLM treats this value as a hard budget for weights + KV + activations + graphs; at 0.99 there is no headroom for the transient activation peak or a co-located process (NCCL, a monitoring agent, another engine). The default is 0.92, and two engines on one GPU should each get roughly half.

Benchmarking decode at batch size 1 and concluding it is compute-bound. Symptom: a benchmark reporting low utilisation, no throughput gain from batching, and a conclusion that “the model is FLOP-limited”. At $B=1$ the intensity is ~1.1 FLOP/byte against a ridge of 295 — you measured the bandwidth wall, not the model. Any decode-optimisation claim needs at least two batch sizes.

Assuming the cache is what makes long context slow. Symptom: you shrink the cache with fp8, see no TTFT improvement, and conclude fp8 “did not work”. Prefill TTFT is compute-bound (the $O(s^2)$ attention term); fp8 KV cache attacks the decode slope and capacity. Different curves.

Turning on prefix caching and assuming it is free. Symptom: throughput unchanged, and under high churn lower than with it off. It only pays when requests genuinely share long prefixes; hashing and block bookkeeping cost real time. It is on by default in vLLM, which is right for chat and wrong for a workload of unique prompts.

Multi-sequence decoding multiplies the cache. n > 1 (parallel sampling) or beams share the prompt’s blocks but diverge afterwards: the PagedAttention paper measures 6.1–9.8% memory saving for parallel sampling on Alpaca and 37.6–55.2% for beam search. Nontrivial, but not “the prompt is free”.

Evicting KV without checking what mattered. Symptom: quality regressions that appear only past the eviction threshold, invisible on short benchmarks. Attention sinks (StreamingLLM) show a few initial tokens absorb disproportionate attention; dropping them can degrade a model far more than the byte count suggests.

6. Using it in a project

Reach for KV-cache work when all three hold: you self-host, your context is long enough that the cache is a meaningful fraction of HBM (roughly: $s\times k$ is a large multiple of a few hundred MB), and your bottleneck is concurrency or TPOT rather than answer quality.

Do not reach for it when:

• Your context is short. Under roughly 2–7k tokens, fp8 KV cache is a loss on ITL (vLLM measures a ~7k break-even), GQA savings are dominated by weight memory, and prefix caching has nothing to share. Measure $sk/2P$ first: if the cache is under ~10% of the weights, this is not your problem.
• Your traffic is low-QPS and single-user. A 5-requests-per-minute internal tool at batch 1 is bandwidth-bound at ~23 tok/s for a 70B whatever you do to the cache, and nobody is queued behind anyone. Buy a smaller model.
• Your bottleneck is elsewhere. If p95 latency is dominated by retrieval, a reranker, a tool call, or output parsing, KV tuning changes a term that is not in the critical path. Profile first; the RAG pipeline is the usual culprit.
• You are paying per token to a provider. Their KV cache is their problem; your levers are prompt length and model choice.
• One GPU already fits the model with room to spare. Paging, quantization, and prefix caching add failure modes to solve a capacity problem you do not have.

Decision order when you do need it: (1) cap max_model_len at the shortest value that serves the product; (2) enable prefix caching if prefixes repeat; (3) switch to a GQA/MLA checkpoint if you control model choice; (4) enable fp8 KV cache if contexts exceed ~7k; (5) only then tune max_num_seqs and batching against an SLO. Steps 1–2 are free and reversible; 3–4 change quality and need evaluation; 5 needs a goodput benchmark, which is the next chapter’s subject.

When in doubt, use retrieval instead of context. The most expensive KV cache is the one holding tokens the model did not need. Four well-chosen chunks beat a 100k window on cost, latency, and usually accuracy.

7. How you would prove it worked

The trap with cache work is optimising a proxy. Prove it on the axis you claimed.

Metric. Claimed capacity → maximum concurrent sequences at the SLO. Claimed speed → TPOT/ITL at fixed concurrency. Claimed throughput → goodput, output tokens per second that still satisfy your TTFT and TPOT SLOs. Never report bare tokens/s; it is trivially inflated by admitting more latency. Always report the baseline — same model, same max_model_len, same arrival trace, same concurrency, change reverted — as a pair.

Ablation. Change one factor at a time; three experiments, one per controllable factor in $M_{KV}$:

1. GQA vs MHA. Use a family that offers both, or uptrain a GQA head on ~5% of tokens per Ainslie et al. Measure resident sequences at a fixed 8k context. Expect $n_{heads}/n_{kv}$; if you see less, something else is binding.
2. fp8 KV on/off at a long context. Fit TPOT against input length (the slope, not one point), measure throughput under load, and measure the accuracy delta on your own task — not only MMLU.
3. Prefix caching on/off on real traffic. Read the cache-hit-rate metric next to throughput; a hit rate near zero explains a null result instantly.

Reporting template (example — replace with your own numbers).

“On 1×H100 with Llama-3.1-70B at a 32k cap, enabling fp8 KV cache raised goodput from 180 to 340 output tok/s (example) at a fixed p95 TPOT of 120 ms, because resident sequences went from 4 to 7 (example) at the same gpu_memory_utilization. Accuracy on our eval set moved from 71.2% to 70.5% (example), inside our 1-point tolerance. Capping max_model_len at 16k would raise concurrency to 12 (example) but breaks two long-document workflows, so we kept 32k and added retrieval for those.”

That report names the SLO, gives both numbers, gives the mechanism, and states the quality cost and the rejected alternative.

8. Interview drill

“Show me the memory arithmetic for serving a 70B at long context.” Strong answer: $M_{KV}=2L{n}_{kv}{d}_{h}sBp$. For Llama-3-70B, $L=80$, $n_{kv}=8$, $d_h=128$, $p=2$ gives 327,680 bytes per token: 2.68 GB per 8k sequence, 42.95 GB per 128k sequence. Weights are ~141 GB in bf16, so the cache equals them at ~53 concurrent 8k sequences but only ~3.3 at 128k. Common wrong answer: quoting the weights as the memory requirement and treating the cache as a rounding error. It is the term that scales with load.

“Why is the factor 2 there, and not 3?” Strong answer: keys and values are both reused by every future token; the query is consumed by its own step. Wrong answer: “query, key, value” — memorised formula, not the dataflow.

“Does GQA reduce the compute or the memory?” Strong answer: primarily memory: it changes the number of stored K/V heads, dividing $n_{kv}$ by the group factor (8× for Llama-3-70B) with no change to $L$, $d_h$, or $s$. It also cuts bytes read per decode step, and value-mixing FLOPs shrink with it, but at long context capacity dominates — it is what lets you admit more sequences. Common wrong answer: “it makes attention cheaper”, conflating capacity, bandwidth, and FLOPs.

“Why is decode memory-bound while prefill is compute-bound?” Strong answer: arithmetic intensity. Prefill serves $n$ tokens per read of the $P$-parameter weights, so its intensity scales with $n$ — ~9,000 FLOP/byte at $n=8192$ for a 70B, 31× above the H100 ridge of 295, so its floor is $2Pn/\text{peak FLOPs}$. Decode reads the same $P$ weights for one token, ~1.1 FLOP/byte and 260× under the ridge, so its floor is bytes over bandwidth: ~43 ms/token. Common wrong answer: “decode is slower because it’s sequential” — sequentiality is a latency property; the FLOP-per-byte ratio sets the bottleneck.

“Can batching make decode compute-bound?” Strong answer: only below ~1,500 tokens of context on an H100. Batching amortises the weight read, but the KV read grows linearly with $B$ at the same rate as the FLOPs, so the asymptotic intensity is $f(s)/(sk)$: ~60 FLOP/byte at 8k, still 4.9× short of the ridge. The ceiling is $\text{BW}/(sk)\approx 1,250$ tok/s regardless of batch. Common wrong answer: “batching always gets there eventually”, or sizing the batch from weights alone and concluding $B\approx 150$ suffices.

“What breaks at 128k context, and what do you do about it?” Strong answer: two different things. Decode: the cache is ~43 GB per sequence and every step reads it, so cap concurrency, use fp8 KV, prefer an MLA/GQA checkpoint; the throughput ceiling falls as $1/s$. Prefill: attention is $O(s^2)$ compute, so TTFT grows quadratically — chunked and disaggregated prefill manage that. Quality is a third question: fitting does not mean the model uses the context well. Common wrong answer: one fix (“quantize the KV cache”) applied to all three; fp8 KV does not help TTFT.

Next: Serving: vLLM and PagedAttention — the cache is this large, so how you allocate it decides your batch size, and batch size decides your throughput.