Part 04 — Inference and Serving

How it is run.

Training decides what the model can do; serving decides whether you can afford to use it. This part is about the two regimes of inference — prefill and decode — and the systems that make production serving affordable: KV-cache management, paged attention with continuous batching, and the distributed execution layer underneath.

1. KV Cache — why autoregressive decoding would otherwise be quadratic, the exact memory formula derived and computed for a real model, and the techniques (GQA, MQA, quantization, eviction) that shrink it.
2. Serving: vLLM and PagedAttention — why the naive per-request KV allocation leaves 60-80% of memory wasted, how paging the cache into fixed blocks behind a block table (plus copy-on-write prefix sharing) fixes it, and what continuous batching does to throughput at fixed latency.
3. Ray — the distributed execution layer most LLM training and serving stacks sit on: tasks vs actors, the object store, placement groups, and Ray Serve. This is the substrate under veRL and OpenRLHF from Part 03.

The single number to carry away: for a 7B model in fp16, the KV cache costs roughly 0.5 MB per token, so a 4096-token conversation is about 2 GB — often more than the weights at realistic batch sizes.

Depends on: Part 03 for the training rollout loop, which runs the same serving machinery this part analyses.

Where this leads: Part 05 puts a control loop around the served model.