Quantization

TL;DR — Quantization stores weights (and optionally activations) in fewer bits: $\hat{w}=s(q-z)$ on an affine grid with step $s$ and zero-point $z$. Autoregressive decoding is memory-bandwidth bound, so time per token is roughly the bytes of weights divided by bandwidth — cutting bits is the most direct throughput lever you have. LLMs are hard to quantize because a handful of outlier channels are ~17× larger than typical values and a single absmax sets the step size for the whole tensor; per-group scaling, outlier-aware decompositions (LLM.int8, SmoothQuant, AWQ) and quantile-placed levels (NF4) are the fixes. 8-bit is essentially free; 4-bit weight-only costs ~1–2% on most tasks and can be much worse on reasoning; below 4 bits you are paying for QAT or rotations.

1. Why this exists

The concrete failure is arithmetic, not accuracy. A 70B model in fp16 is $70\times 10^9\times 2=140$ GB of weights. That does not fit on one 80 GB accelerator, so you are immediately into multi-GPU tensor parallelism — or a quantized checkpoint that fits.

The second failure is slower and more interesting: even when the model fits, decoding is bandwidth-bound. An autoregressive step generates one token and must read every weight matrix once. So

$$t_{\text{token}}\approx \frac{\text{bytes of weights read}}{\text{memory bandwidth}},$$

with no dependence on the FLOP count until batch size grows enough to make the matmul compute-bound. On an H100 (3.35 TB/s HBM3):

Those numbers are the ceiling — they ignore the KV cache, attention, and kernel inefficiency — but they set the agenda. Halving the bits halves the bytes and roughly doubles your decode ceiling. Name the axis: quantization is a memory-and-bandwidth optimisation, not a FLOPs optimisation. (And it is a different axis from the KV-cache problem, which is covered in KV Cache.)

The third failure is the reason the rest of this chapter is long: the naive version of this — round every weight to the nearest of 16 evenly spaced levels — destroys LLM quality, because a small number of outlier values dominate the range.

2. How it works

2.1 The affine quantiser, symbol by symbol

Mapping a real weight $w$ to a $b$-bit integer and back:

$$q=\mathrm{clamp}\left(\mathrm{round}\left(\frac{w}{s}\right)+z,q_{\min },q_{\max }\right),\hat{w}=s(q-z).$$

• $b$ — the bit width: 8, 4, 3, or 2. It fixes the number of representable levels at $2^b$ and therefore the resolution floor. Nothing else in the formula can recover resolution you did not buy; a smaller $b$ is a strictly coarser grid, which is why 3-bit fails where 4-bit succeeds.
• $q_{\min },q_{\max }$ — the integer range of the grid. Signed symmetric INT4 uses $[-8,7]$ ($q_{\min }=-2^{b-1}$, $q_{\max }=2^{b-1}-1$); unsigned INT4 or INT8 uses $[0,2^b-1]$. The asymmetry of the signed range is not a typo: two’s complement gives one more negative level, and by convention zero is placed inside the range.
• $s$ — the scale, the real value of one integer step. It is what converts bits into magnitudes, and it is the single most important quantity in the whole chapter. For an asymmetric grid over the observed range, $s=(w_{\max }-w_{\min })/(q_{\max }-q_{\min })$, and for a symmetric grid over the absmax, $s=\max |w|/(2^{b-1}-1)$. Smaller $s$ means finer resolution but a narrower representable range; if $s$ is too small, large values clip. Every accuracy problem in this chapter is a problem of $s$ being too large, because it was computed from an extreme value rather than from the bulk of the distribution.
• $z$ — the zero-point, the integer that represents $w=0$: $z=\mathrm{round}(q_{\min }-w_{\min }/s)$. It shifts the grid so that exactly-zero weights are representable with no error. Drop it (force $z=0$, i.e. symmetric quantization) and the grid is no longer centred on the data: for a distribution whose mass sits just above zero, every near-zero weight is pushed to a nonzero level, which injects a constant offset into every dot product. That is why activations after a non-linearity — ReLU or SwiGLU outputs, which are one-sided and heavily skewed — almost always use asymmetric grids, while weight matrices, which are roughly zero-mean, use symmetric ones. Padding tokens and zeroed activations also need an exact zero, which is the same argument from the other direction.
• $\mathrm{round}(\cdot )$ — round-to-nearest, which bounds the error at $|w-\hat{w}|\le s/2$. This bound is worth holding onto: it says the entire accuracy question reduces to how small you can make $s$ without clipping.
• $\mathrm{clamp}(\cdot )$ — saturates values outside the grid instead of wrapping them. Drop it and an outlier overflows into a wrong level in the opposite direction — a large positive weight silently becomes a large negative one, which is far worse than clipping.
• $\hat{w}$ — the dequantised weight. Note the important structural fact: after quantising, everything downstream multiplies in a normal float dtype. INT4 on current GPUs is not usually an INT4 multiply; it is a dequantise-to-fp16-then-GEMM, or a packed special-case kernel. That distinction returns in §5.

Worked example showing why a single outlier ruins a per-tensor grid (example): a weight tensor whose typical entries are $\pm 0.1$ contains one value at $60$. Per-tensor symmetric INT4 gives $s=60/7=8.57$. The step is 8.57, so a typical weight of $0.1$ rounds to $q=0$ and dequantises to $0.0$: the outlier has turned the entire tensor into zeros, with a relative error of 100% on every entry that matters.

2.2 Granularity is the fix, and it has an exact price

If one scale per tensor is too coarse, use one per group. Partition the tensor into contiguous groups of $g$ values, each with its own $s$ (and $z$ for asymmetric grids):

$$s_g=\frac{{\max }_{i\in g}{w}_i-{\min }_{i\in g}{w}_i}{q_{\max }-q_{\min }}.$$

• Per-tensor ($g=$ everything) — one scale for the whole matrix. Cheapest, worst.
• Per-channel / per-row — one scale per output row. This is what GPTQ uses (“uniform per-row asymmetric quantization on the min-max grid”, Frantar et al. 2023 §5), and it matches the fact that outlier channels are a per-dimension phenomenon. Cost: one scale per row, negligible.
• Per-group / per-block ($g$ = 32–256) — the LLM serving default, and the parameter you actually tune. Smaller $g$ means finer resolution and more scale bytes.

The price is exact and easy to quote. With an fp16 scale and $g$ weights per group, the overhead is $16/g$ bits per parameter:

The third column is a heuristic (example): for Gaussian weights the expected group maximum is $\max |w|\approx \sigma \sqrt{2\ln g}$, so the step relative to $\sigma$ is $\sqrt{2\ln g}/7$. Read it as a shape, not a precise number: halving the group size from 128 to 64 buys you ~7% smaller steps for +0.125 bits/param; going to 32 buys ~15% for +0.375 bits/param. Diminishing returns set in fast, which is why 64 or 128 is the universal default and why the QLoRA authors chose block size 64 for weights.

The other thing granularity buys is containment. In the outlier example above, per-tensor scaling destroyed all 16.7 M weights of a $4096\times 4096$ matrix; with $g=128$ only the 128 weights in the outlier’s own group are affected — 0.0008% of the matrix. That is the entire argument for blocked quantization in one calculation.

2.3 Why LLMs are specifically hard: activation and weight outliers

Small models quantize to 4 bits without drama. LLMs do not, and the reason is measurable systematic outliers. The transformers documentation for LLM.int8() summarises the finding: values “are usually normally distributed, that is, most values are in the range [-3.5, 3.5], but there are some exceptional systematic outliers that are very differently distributed for large models. These outliers are often in the interval [-60, -6] or [6, 60]” (Dettmers et al. 2022, quoted in the BitsAndBytesConfig docs). The ratio is about $60/3.5\approx 17\times$ — and $s$ is set by the extreme value, not the typical one, so the ~17× outlier costs you ~17× of your 4-bit resolution on everything else.

Three families of fixes, all of which you should be able to name:

Keep the outliers in high precision (LLM.int8()). Detect the feature dimensions whose activation magnitude exceeds a threshold and compute those columns in fp16, everything else in INT8 — a mixed-precision decomposition. The threshold is exposed as llm_int8_threshold, default 6.0, which is exactly the lower edge of the observed outlier band. This is why 8-bit LLM inference was “free” years before 4-bit worked.

Move the difficulty into the weights (SmoothQuant). Activations quantize badly because of a few large channels; weights quantize easily. Insert a per-channel diagonal scaling:

$$Y=(X\mathrm{diag}(s{)}^{-1})(\mathrm{diag}(s)W),s_j=\frac{{\max }_i|X_{ij}{|}^{\alpha }}{{\max }_i|W_{ij}{|}^{1-\alpha }}.$$

Every symbol: $X$ and $W$ are the layer’s input and weight; $\mathrm{diag}(s{)}^{-1}$ and $\mathrm{diag}(s)$ cancel exactly, so this is an algebraic identity that changes nothing mathematically. What it changes is which tensor is hard. $\alpha \in [0,1]$ is the migration strength: at $\alpha =1$ the scale is $s_j={\max }_i|X_{ij}|$, so every activation column is divided down into $[-1,1]$ — the activations become easy and the difficulty lands on the weights; at $\alpha =0$ it is the reverse, the weights are made easy and the activations keep their outliers. The paper’s default ~0.5 splits it. ${\max }_i|X_{ij}|$ is the per-channel activation range that caused the trouble and ${\max }_i|W_{ij}|$ is the weight range that can absorb it. Remove the scaling and you are back to plain W8A8, which loses accuracy on the activation side.

Protect the salient channels (AWQ). AWQ’s observation is that “not all weights in an LLM are equally important. Protecting only 1% salient weights can greatly reduce quantization error”, and that salience must be read off the activation distribution rather than the weights. It identifies those channels from offline activation statistics and scales them up before quantization, which (its derivation shows) reduces their relative error. No backpropagation and no reconstruction means it generalises across domains better than calibration-fitting methods (Lin et al. 2023, abstract). Its companion runtime, TinyChat, reports >3× speedup over the Hugging Face fp16 implementation on desktop and mobile GPUs.

One sentence to remember. Outliers, not bit width, are the reason LLM quantization is a research field; every method above is a different way of stopping one large value from setting the grid for everything else.

2.4 NF4 and the quantile assumption

NF4 takes a different route: instead of choosing where to put the levels adaptively, it chooses them once, from a distributional assumption.

The levels are the midpoints between adjacent quantiles of a standard normal:

$$q_i=\frac{1}{2}\left(Q_X\left(\frac{i}{2^k+1}\right)+Q_X\left(\frac{i+1}{2^k+1}\right)\right),i=0,1,\dots ,2^k-1,$$

where $Q_X$ is the quantile function (inverse CDF) of $\mathcal{N}(0,1)$ and $k$ is the bit width. Name the pieces and the reasoning:

• $Q_X(p)$ — the value below which a fraction $p$ of a standard normal lies. It is the tool that converts “equal probability mass” into “a number on the real line”.
• The interval between $Q_X(i/(2^k+1))$ and $Q_X((i+1)/(2^k+1))$ contains exactly $1/(2^k+1)$ of the probability mass. Taking the midpoint of that interval is the value that minimises the squared error contribution of that bin. So each of the $2^k$ levels carries the same expected number of weights, and the grid is optimal for samples drawn from $\mathcal{N}(0,1)$. This is the “information-theoretically optimal for normally distributed data” claim in the QLoRA abstract.
• The $2^k+1$ in the denominator (rather than $2^k$) is the offset that makes the quantile boundaries and the level midpoints line up.

Two mechanical details complete the format:

• Exact zero. A symmetric quantile grid has no level at exactly 0, and zero is both the mode of the weight distribution and the value of padding. QLoRA’s construction therefore estimates $2^{k-1}$ levels for the negative half and $2^{k-1}+1$ for the positive half, unifies them and drops one of the two duplicate zeros. That is the “asymmetric data type” in the paper.
• Block absmax rescaling comes first. Before mapping to the fixed grid, each block of 64 weights is divided by its own $\max |w|$, landing it in $[-1,1]$. So NF4 is a fixed non-uniform codebook plus a per-block scale — the quantile assumption supplies the codebook, granularity supplies the scale.

The assumption, and what breaks when it is false. The whole argument rests on weights being approximately zero-mean Gaussian (the paper verifies this empirically in its Appendix F). If a block is not — heavy tails, or a channel with a few huge entries — then absmax rescaling maps the extreme value to $\pm 1$, where the Gaussian’s levels are sparse (there is very little mass near $\pm 1$), while the bulk of the block gets over-resolved levels it did not need. The error concentrates in exactly the largest weights. This is not hypothetical: it is why NF4 is used at block size 64 rather than per-tensor, why double quantisation exists at all, and why the paper still needs LoRA adapters on every layer to recover 16-bit quality rather than relying on the format alone.

The empirical payoff, from the QLoRA paper’s own comparison on Pile Common Crawl mean perplexity across 125M–13B models (Dettmers et al. 2023, Table 2):

Note what this says: NF4 beats a 4-bit float, despite FP4 having an exponent that gives it a much wider dynamic range. The reason is the shape of the levels — FP4’s E2M1 values are $\{0,0.5,1,1.5,2,3,4,6\}$, which spends resolution on the tails, whereas NF4 spends it where the mass is. A wide dynamic range is not the same as low error on the distribution you actually have.

2.5 PTQ versus QAT

Post-training quantization (PTQ) quantizes a trained model with no gradients. It may use a small calibration set to estimate activation ranges, but nothing is trained. This is the default for LLM serving because it costs minutes to hours, and it is what every format in §2.6 is. Two sub-families, and you should always say which one you mean:

• Weight-only (W4A16) — 4-bit weights, 16-bit activations and matmuls. The memory win is real; the compute is still fp16. This is what GPTQ, AWQ and bitsandbytes NF4 do.
• Weight + activation (W8A8, W4A8, W4A4) — activations are quantized too, which is what unlocks the fast integer/FP8 tensor-core paths. It is harder because activations are dynamic and outlier-ridden, requiring SmoothQuant-style migration or rotation.

Quantization-aware training (QAT) simulates the rounding in the forward pass — the straight-through estimator passes the gradient through round as if it were the identity — so the model adapts to the grid it will be deployed on. It costs a training run, so reach for it only when PTQ has demonstrably failed: sub-4-bit weights, or W4A4 without a rotation. See Esser et al. 2020 and Liu et al. 2023.

2.6 The format zoo

flowchart TB
    M["Trained fp16/bf16 model"] --> CHOICE{"What is the bottleneck?"}
    CHOICE -->|"GPU decode, NVIDIA"| K{"Weight-only or W8A8?"}
    CHOICE -->|"CPU / Apple silicon"| GGUF["GGUF k-quants / i-quants"]
    CHOICE -->|"Hopper/Ada/Blackwell"| FP8["FP8 W8A8"]
    CHOICE -->|"Blackwell, 4-bit"| MX["MXFP4 / NVFP4"]
    K -->|"weight-only"| GPTQ["GPTQ / AWQ / NF4, needs Marlin-class kernel"]
    K -->|"activations too"| SQ["SmoothQuant / FP8"]

GPTQ in one paragraph, because the mechanism is the interesting part. It solves a layer-wise reconstruction problem, $\arg {\min }_{\hat{W}}\Vert WX-\hat{W}X{\Vert }_2^2$, rather than rounding each weight independently. The trick inherited from Optimal Brain Quantization is that this objective is quadratic with Hessian $H_F=2X_F{X}_F^{\top }$, where $F$ is the set of not-yet-quantized weights and $X_F$ the corresponding layer inputs. so when you round a weight and incur error $w-\mathrm{quant}(w)$, you can compute the optimal compensating update to all remaining weights in closed form. GPTQ’s contributions are the engineering ones that make this scale: a fixed column order instead of greedy order (so $H^{-1}$ is updated once per column instead of once per weight), lazy batch updates of 128 columns at a time, and a Cholesky reformulation to keep $H^{-1}$ positive definite at scale. The result quantizes OPT-175B in 4.2 GPU-hours on a single A100 with 3–4-bit weights (Frantar et al. 2023). Accuracy: on OPT-175B, 4-bit GPTQ loses 0.03 perplexity versus fp16 while round-to-nearest loses 2.2; at 3-bit round-to-nearest “collapses completely” and GPTQ still produces reasonable perplexity.

GGUF effective bits — derive them, do not guess them. A “4-bit” GGUF file is not 4.0 bits per weight, and comparing checkpoints by their nominal width is a classic mistake. The ggml block layouts make the real number computable (ggml-common.h, with super-block size QK_K = 256):

Read the Q4_K row as the general lesson: 4 bits of mantissa, plus 12 bytes of 6-bit scales/mins and 4 bytes of fp16 super-block scales per 256 weights, is 4.5 bits per stored parameter. Q6_K is 6.5625, not 6. This table is why “Q4_K_M” appears in memory estimates as ~4.5 bpw.

FP8 is the modern default on new accelerators because it needs no calibration to be useful. E4M3 (4 exponent bits, 3 mantissa) is used for weights and activations; E5M2 trades mantissa for exponent range and is used for gradients, which span a wider range. FP8 tensor cores exist on Hopper (H100/H200), Ada (L4/L40S/RTX 40) and Blackwell; Ampere (A100) has no FP8 support. DeepSeek-V3 reported FP8 mixed-precision training at scale (arXiv:2412.19437). In transformers, fine-grained FP8 uses FineGrainedFP8Config with a weight block size of (128, 128) by default.

MXFP4 and NVFP4 are the OCP Microscaling and NVIDIA answers to “4 bits with a shared scale”. MXFP4 shares one E8M0 power-of-two scale across 32 values; NVFP4 shares a higher-precision E4M3 scale across 16 values plus a second-level FP32 scale per tensor. The finer scale encoding is the point: NVIDIA shows E4M3 scaling achieving lower mean-squared error than E8M0’s power-of-two snapping, and halving the block from 32 to 16 gives twice as many chances to match the local dynamic range. NVIDIA claims NVFP4 comes within ~1% of FP8 on DeepSeek-R1-0528 at ~3.5× less memory than FP16 (NVIDIA, 2025). Both formats cost 4.25–4.5 effective bits per weight, consistent with the ggml table above.

3. Implementation

4-bit inference with bitsandbytes through transformers. Note which of these are defaults you are overriding [as of 2026-10]; the transformers defaults are bnb_4bit_quant_type="fp4" and bnb_4bit_compute_dtype=torch.float32, neither of which is what you want.

# bitsandbytes==0.50.2, transformers (2026)
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # default is "fp4"; NF4 is the better 4-bit codebook
    bnb_4bit_use_double_quant=True,         # default False; costs ~0, saves ~0.37 bits/param
    bnb_4bit_compute_dtype=torch.bfloat16,  # default torch.float32; this is the *compute* type
    bnb_4bit_quant_storage=torch.uint8,     # default; use bfloat16 if you need FSDP sharding
)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-14B", quantization_config=bnb, device_map="auto"
)

8-bit with the outlier decomposition, and the two knobs that matter:

bnb8 = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,          # default; the outlier band is [-60,-6] ∪ [6,60]
    llm_int8_skip_modules=["lm_head"],   # never quantize the output head
)

A quantized checkpoint served by vLLM — the production path. Supported backends [as of 2026-10] include AWQ, GPTQModel, BitsAndBytes, FP8, INT4 W4A16, INT8 W8A8, GGUF, TorchAO and AMD Quark:

vllm serve Qwen/Qwen3-32B-AWQ       --quantization awq       --max-model-len 8192
vllm serve <fp8-checkpoint>         --quantization fp8
vllm serve <bnb-4bit-checkpoint>    --quantization bitsandbytes

GGUF is the CPU / Apple-silicon path, and llama.cpp will fetch and run a pre-quantised file directly:

llama cli   -hf ggml-org/Qwen3.5-8B-GGUF:Q4_K_M   # interactive
llama serve -hf ggml-org/Qwen3.5-8B-GGUF:Q4_K_M   # OpenAI-compatible server
# produce your own quant with the bundled llama-quantize utility [as of 2026-10]

Measure with a harness rather than a vibe:

lm_eval --model hf \
        --model_args pretrained=Qwen/Qwen3-8B,dtype=bfloat16 \
        --tasks mmlu,gsm8k --batch_size 8
# then repeat with the quantized checkpoint and diff the per-task, not just mean, numbers

Knobs that matter, with realistic ranges:

4. Numbers and trade-offs

Memory, derived from the effective bits in §2.6 (decimal GB):

Two corrections that catch people out:

• Nominal 4-bit is not 4.0 bpw. “7B at 4-bit = 3.5 GB” ignores the 0.127 bits of double-quantised scales and, for GGUF, the ~0.5 bits of block metadata. Budget 3.6–3.9 GB.
• The KV cache is not in this table. At the sequence lengths that matter it is not small. Recompute it — see KV Cache — before you promise a memory number.

Throughput, using the bandwidth ceiling from §1: quantization buys you roughly raw_bits / quant_bits on decode, i.e. ~3.5× for 4-bit versus fp16 on the weight-read term. That is the whole win, and it is an upper bound. It does not apply to prefill, which is compute-bound and barely cares about weight bits.

Accuracy, with real measured numbers rather than adjectives:

The honest summary: 8-bit is essentially free; 4-bit weight-only costs about 1–2% on average benchmark scores and rather more on reasoning, math and long-context tasks, where the damage concentrates; 3-bit is risky; below that you need rotations (QuaRot, Ashkboos et al. 2024; SpinQuant, Liu et al. 2024) or QAT.

Which axis you are optimising:

5. Pitfalls

Quantizing the wrong layers. Symptom: quality collapses far beyond what the format should cost — the model babbles rather than degrades gracefully. Diagnosis: quantizing lm_head and the embedding matrix, or the first and last transformer blocks, which are known to be the most sensitive. The output head is one linear layer that maps directly to the logits, so its error lands undiluted on the probability distribution. Fix: llm_int8_skip_modules=["lm_head"], keep embed_tokens in fp16, and consider leaving the first block unquantized.

Sizing for weights only. Symptom: the model loads, serves a few short requests, then OOMs when a long prompt arrives. Diagnosis: the KV cache is not in the quantized-weight number and grows linearly with sequence length and batch. Fix: compute it explicitly; consider quantized KV cache (--kv-cache-dtype fp8 and friends) as a separate decision from weight quantization.

Measuring only perplexity. Symptom: your eval says “0.1 perplexity worse, ship it”, and users report the model is worse at arithmetic and code. Diagnosis: perplexity is dominated by the bulk of easy next-token predictions and is insensitive to the long tail where quantization damage lives. Fix: always run per-task accuracy — MMLU, GSM8K, long-context retrieval — and look at which tasks moved, not the mean.

Expecting a speedup on hardware without 4-bit kernels. Symptom: 4-bit uses less memory and is slower per token, especially at small batch or long sequence. Diagnosis: without a dedicated kernel (Marlin, ExLlama, Machete), INT4 is dequantise-to-fp16 plus a normal GEMM, which adds work instead of removing it. GPTQ’s own paper notes its method “does not provide speedups for the actual multiplications, due to the lack of hardware support for mixed-precision operands”; the 3.25× it reports comes from memory loading with bespoke kernels. Fix: on Ampere/Ada/Hopper use a Marlin-class kernel; on Blackwell use native MXFP4/NVFP4; on CPU use GGUF, which is genuinely optimized for that path.

Calibrating on the wrong distribution. Symptom: the quantized model scores like fp16 on generic text but is noticeably worse on your domain. Diagnosis: GPTQ and AWQ pick scales from the calibration set’s activation ranges; generic web text does not produce the activation statistics of legal filings or code. Fix: calibrate on a sample of real in-domain prompts, and prefer AWQ when in doubt because it explicitly avoids overfitting the calibration set.

A checkpoint/loader configuration mismatch. Symptom: fluent-looking garbage, or NaNs — not a mild quality drop. Diagnosis: group size, symmetry or bits in the checkpoint do not match what the loader assumes. Fix: always load quantized checkpoints with the config that shipped with them, and sanity-check with a fixed prompt after loading.

Merging a LoRA adapter into a quantized base. Symptom: the merged model is worse than base-plus-adapter by more than round-off. Diagnosis: the adapter was trained against the quantized weights; $\mathrm{quant}(W_0+\Delta W)$ is not $\mathrm{quant}(W_0)+\Delta W$. Fix: merge into a 16-bit base and re-quantize, or serve unmerged — cross-referenced in LoRA and QLoRA.

6. Using it in a project

Quantize when memory or decode bandwidth is the binding constraint and you have measured that quality holds on your task. That is a narrower condition than it sounds.

When NOT to reach for quantization:

• The model already fits and latency is fine. You are then trading real accuracy risk for a resource you were not short of. Measure before optimising.
• You are compute-bound (large-batch prefill). Weight quantization does almost nothing for FLOPs; you need activation quantization (W8A8) or a better kernel. Do not expect a large-batch throughput win from a 4-bit weight-only checkpoint.
• Your hardware has no INT4/FP8 path. On an A100 with a 4-bit checkpoint and no Marlin-class kernel you can end up slower than fp16. On older consumer GPUs, same story.
• You are about to fine-tune and then merge. The QLoRA merge path is lossy; if you will fine-tune, plan for serving base-plus-adapter or a 16-bit merge, and read LoRA and QLoRA.
• The task is at the edge of the model’s ability. Reasoning and math degrade first. If your product depends on the model being right rather than fluent, prefer 8-bit or fp16 — or a smaller fp16 model, if the benchmarks say so.

When it is clearly right: serving a model too large for your GPU (§1); fitting a bigger model on the same hardware, which is often a better trade than a smaller model at higher precision — the QLoRA paper’s conclusion is that “with a given finetuning and inference resource budget it is beneficial to increase the number of parameters in the base model while decreasing their precision”; and CPU/edge deployment, where GGUF’s k-quants are the mature answer.

The decision in one pass:

1. Does fp16 fit, with the KV cache, at your target batch and context? → ship fp16, stop.
2. Is it a GPU deployment with an outlier-heavy workload? → INT8/FP8 first; it is nearly free.
3. Still too big? → 4-bit weight-only (AWQ or GPTQ for a GPU, Q4_K_M for CPU), keeping lm_head and embeddings in high precision.
4. Quality drop on your eval? → measure per task; try group size 64, AWQ over GPTQ, or NVFP4 on Blackwell.
5. Still failing? → rotation-based W4A4, or QAT — and only then.

7. How you would prove it worked

Baseline is the uncompressed fp16 model on the same eval, same prompts, same sampling parameters. Not the model card’s numbers, and not the quantized model’s own training loss.

Four measurements, in order of how often they catch real regressions:

1. Per-task accuracy with lm-evaluation-harness — MMLU, GSM8K, ARC, HellaSwag — plus one long-context task if you have long contexts. Report a table of per-task deltas, not a mean, because the mean hides exactly the reasoning regressions that matter.
2. In-domain held-out quality. (example) A task metric on 200 labelled examples from your actual traffic is worth more than any public benchmark.
3. KL divergence between the quantized and fp16 next-token distributions on a held-out sample. Cheap, and sensitive to subtle distribution shift that accuracy hides.
4. Latency and throughput at your real batch/sequence mix. Time-to-first-token (prefill, compute-bound) and inter-token latency (decode, the bandwidth-bound number from §1) separately, because quantization helps one and not the other much.

Reporting template (example — replace with your own numbers). “Quantized Qwen3-32B to AWQ 4-bit, group size 128. Weight memory 65 GB → 18 GB; inter-token latency 48 ms → 15 ms at batch 1 on one H100, with no change in time-to-first-token at 4k prefill. MMLU 74.1 → 73.6, GSM8K 71.2 → 68.9 — a 2.3-point reasoning regression — so we shipped 8-bit instead (MMLU 73.9, GSM8K 70.8, 33 GB) and kept 4-bit for the batch-8 job, where the throughput win is 3.1×.”

The one ablation worth running: round-to-nearest versus your method, at the same bit width and granularity. If the quantizer you chose does not beat naive RTN, you are paying complexity for nothing — and on smaller models GPTQ is competitive with much more expensive methods precisely because the reconstruction objective, not the kernel, does the work.

8. Interview drill

“What does the zero-point do, and when do you set it to zero?” Strong answer: it is the integer that encodes $w=0$, so that the grid can be placed over a range that does not contain zero, or is skewed relative to it. Activations after a one-sided non-linearity need it; weights, which are roughly zero-mean, do not, so symmetric quantization is used and you save the extra parameter. Padding tokens and zeroed activations also require an exact zero, which is the same constraint from the other side. Common wrong answer: “it shifts the values so they’re all positive.” It shifts the grid mapping, not the data, and there is no requirement that quantized values be positive.

“Why is 4-bit hard for LLMs when 8-bit is free?” Strong answer: because $s$ is set by the extreme value of the group, and LLM activations and weights contain systematic outliers roughly 17× larger than typical values (the observed band is $[-60,-6]\cup [6,60]$ against a typical $\pm 3.5$). At 8 bits you have enough levels that the outlier costs little; at 4 bits you have 15 magnitudes and the outlier eats the range. The fixes are granularity (per-group scales), outlier-aware methods (LLM.int8, SmoothQuant, AWQ) and codebooks shaped to the data (NF4). Common wrong answer: “4 bits isn’t enough precision for 16 levels.” Sixteen levels is fine for a well-scaled distribution; the problem is where the 16 levels sit.

“How is NF4 different from INT4?” Strong answer: INT4 places 16 evenly spaced levels in $[-1,1]$; NF4 places them at the midpoints of the quantiles of $\mathcal{N}(0,1)$, so each level carries equal probability mass under the assumption that weights are zero-mean Gaussian. It is a fixed codebook plus a per-block absmax scale, with an asymmetric construction that gives an exact zero. The assumption is load-bearing: for heavy-tailed blocks, absmax rescaling pushes the largest values to $\pm 1$ where the Gaussian levels are sparse. Follow-up: “why does NF4 beat FP4, which has more dynamic range?” → because FP4 spends its resolution on the tails; a wide range is not the same as low error on the actual distribution.

“QLoRA, GPTQ, AWQ — which would you pick and why?” Strong answer: they are not competitors; they answer different questions. NF4/QLoRA is the training format — it lets you fine-tune a 4-bit frozen base with 16-bit adapters. GPTQ and AWQ are inference formats: GPTQ minimizes a layer-wise reconstruction error with second-order information and needs a calibration set; AWQ identifies the ~1% of channels that matter from activation statistics and scales them, without backpropagation, which generalises better across domains. For a GPU serving deployment with no fine-tuning I would start with AWQ; if I needed W8A8 I would use SmoothQuant or FP8. Common wrong answer: treating QLoRA as an inference quantization scheme.

“Does 4-bit quantization make inference 4× faster?” Strong answer: memory drops 4× (plus overhead, so ~3.6×) and, because decode is bandwidth-bound, the ceiling on inter-token latency improves by roughly that factor. The realized speedup depends on kernel support: without a Marlin-class kernel, INT4 is dequantise-then-GEMM and can be slower than fp16. Prefill is compute-bound and barely improves at all. So: yes on the memory axis, sometimes on decode latency, no on prefill and no on FLOPs. Common wrong answer: “4× on everything”, or “no speedup ever”.

“You quantized to 4-bit: perplexity is fine but math and code are much worse. What do you do?” Strong answer: treat it as expected, not as a bug — quantization damage concentrates on reasoning and long-context tasks while perplexity stays flat. Escalate in this order: move to 8-bit or FP8 (usually nearly free); if memory forces 4-bit, switch to AWQ or NVFP4, shrink the group size to 64, and keep lm_head, embeddings and the first block unquantized; if that still fails, the honest answer is that this model cannot afford 4-bit and you should either serve fp16 or use a smaller model. Common wrong answer: re-running the quantizer with a different seed, or blaming the calibration set before checking whether 8-bit would simply solve it.

“When must you pay for QAT?” Strong answer: when PTQ has measurably failed at the bit width you need — typically sub-4-bit weights, or 4-bit weights and 4-bit activations (W4A4) without a rotation method. QAT simulates rounding in the forward pass with a straight-through estimator so the weights adapt to the grid. Before paying that cost, try rotation-based PTQ (QuaRot, SpinQuant), which spreads outliers with an orthogonal transform and is far cheaper. Common wrong answer: “QAT is always better, so use it.” It costs a full training run and is unnecessary at 8-bit, where PTQ is effectively lossless.

Next: RLHF: The Three Stages — SFT gives you a model that follows instructions, and the previous two chapters gave you the machinery to train and serve it cheaply; RLHF is what turns that into a model that follows preferences, using the same frozen-base-plus-adapter trick to keep the policy on one GPU.