LoRA and QLoRA

TL;DR — Full fine-tuning stores a whole model and its optimizer state for every task, so a 7B model needs roughly 12–16 bytes per parameter of training memory. LoRA freezes $W_0$ and trains a rank-$r$ correction $\Delta W=\frac{\alpha }{r}BA$, which cuts trainable parameters to a fraction of a percent and removes optimizer state for the base. $B$ is initialised to zero so the frozen model is exactly the pretrained model at step 0; $A$ is random so $B$ actually receives gradient. The $\alpha /r$ divisor is what stops the update magnitude from growing with $r$. QLoRA keeps the same adapters but stores the frozen base in 4-bit NF4 with double quantisation, so a 7B fine-tune fits on a 16 GB card at almost no measured quality cost.

1. Why this exists

The failure is not “fine-tuning is slow”. It is that the memory cost of a full fine-tune is set by the optimizer, not the model, and it lands with a factor of 8 on top of the weights.

AdamW in mixed precision holds, per parameter:

That is 16 bytes per parameter, or $7\times 10^9\times 16=112$ GB for a 7B model — before a single activation. Drop the master weights and you are still at 12 bytes/param = 84 GB. And the ceiling moves fast with scale: the QLoRA paper opens by noting that 16-bit fine-tuning of a LLaMA 65B parameter model requires more than 780 GB of GPU memory (Dettmers et al. 2023, §1).

Two more things push toward adapters rather than full fine-tuning:

• Storage per task. A full fine-tune is a $d^2$-sized artifact per task. Ten customers means ten 14 GB checkpoints. An adapter is tens of megabytes.
• Forgetting. A full fine-tune moves $W_0$ everywhere. LoRA cannot move $W_0$ at all — the base is frozen and only the additive correction is learned, which is a structural (not incidental) forgetting guard. “LoRA Learns Less and Forgets Less” (Biderman et al. 2024) measures exactly that trade: on code and math LoRA trails full fine-tuning, but it preserves out-of-domain performance better.

Where this matters for you. The interview question is never “what is LoRA”. It is “why does LoRA cost 18 GB when the model is 14 GB, and what would full fine-tuning have cost?” The answer is the table above and the arithmetic in §4.

2. How it works

2.1 The low-rank update

For a frozen pretrained weight $W_0\in {\mathbb{R}}^{d_{out}\times d_{in}}$, LoRA learns

$$W=W_0+\Delta W=W_0+\frac{\alpha }{r}BA,B\in {\mathbb{R}}^{d_{out}\times r},A\in {\mathbb{R}}^{r\times d_{in}}$$

and the forward pass for an input row $x\in {\mathbb{R}}^{d_{in}}$ becomes

$$h=W_{0}x+\frac{\alpha }{r}B(Ax).$$

Every symbol earns its place:

• $W_0$ — the frozen pretrained matrix. It is not updated and, critically, carries no gradient and no optimizer state. That is where the memory saving comes from; unfreeze it and you have full fine-tuning with extra steps.
• $A\in {\mathbb{R}}^{r\times d_{in}}$ — the down-projection, mapping the $d_{in}$-wide input into an $r$-dimensional bottleneck. Remove $A$ and $B$ has nothing to act on.
• $B\in {\mathbb{R}}^{d_{out}\times r}$ — the up-projection, mapping the bottleneck back out. The product $BA$ is the actual update; neither factor is meaningful alone, which is why the pair is the trainable unit.
• $r$ — the rank, and the reason this is cheap: $BA$ has at most $r$ nonzero singular values while $W_0$ has $\min (d_{in},d_{out})$. It is the only real capacity knob — the number of independent directions in which the layer may change. The LoRA hypothesis is that a task’s change has far lower rank than the weight itself; where that is false (new knowledge, new scripts) LoRA underperforms, see §6.
• $\alpha$ — a scaling constant on the update before it meets $W_0$. It changes how large the correction is relative to the frozen weights, which is why it interacts with the learning rate; it is not capacity.
• $\alpha /r$ — the effective scale. Explained in §2.3; this, not $\alpha$, is what the forward pass multiplies by.

flowchart LR
    X["x (d_in)"] --> W0["W_0 x (frozen, no grad)"]
    X --> A["A x (r)"]
    A --> B["B(A x) (d_out)"]
    B --> S["x alpha/r"]
    W0 --> SUM["h = W_0 x + (alpha/r) B A x"]
    S --> SUM

At inference the two branches collapse: $W_{\text{merged}}=W_0+\frac{\alpha }{r}BA$ is a $d_{out}\times d_{in}$ matrix like any other, so there is zero added latency once merged. This is the property that separates LoRA from adapter-layer methods (Houlsby et al.) and from prompt tuning, which pay a permanent inference cost.

2.2 Why $B$ is zero-initialised — and what breaks otherwise

This is the question the stub asks, and the answer has two halves: a forward argument and a gradient argument. Both matter.

Forward: the model must be a no-op at step 0. With $B=0$, $\Delta W=BA=0$ exactly, so $W=W_0$ and the loss at step 0 is bit-for-bit the pretrained model’s loss. Training therefore starts on the pretrained solution and can only improve from it. If you instead initialise both $A$ and $B$ randomly, $\Delta W$ is a random dense perturbation with magnitude comparable to the weights themselves; the first few hundred steps are spent undoing damage to 32 layers of pretrained features, and on a small dataset that damage is often not fully undone.

Gradient: $A$ must be nonzero or nothing moves. Let $g=\partial \mathcal{L}/\partial h$ be the gradient arriving at this layer’s output. The chain rule through $h=W_{0}x+\frac{\alpha }{r}B(Ax)$ gives

$$\frac{\partial \mathcal{L}}{\partial B}=\frac{\alpha }{r}g(Ax{)}^{\top },\frac{\partial \mathcal{L}}{\partial A}=\frac{\alpha }{r}{B}^{\top }g{x}^{\top }.$$

Read those two expressions:

• $B=0$, $A$ random. $\partial \mathcal{L}/\partial B\propto g(Ax{)}^{\top }\ne 0$ as long as $Ax\ne 0$ — so $B$ gets a gradient on step 1 and becomes nonzero. $\partial \mathcal{L}/\partial A\propto B^{\top }g{x}^{\top }=0$ — so $A$ gets no gradient on step 1. That is fine: by step 2, $B\ne 0$, and $A$ starts learning. The asymmetry is the design. Training is effectively “fit the output direction first, then refine the input direction”.
• $A=0$ and $B=0$ (both zero). Both gradients are zero. The pair sits on a saddle and never leaves it, regardless of how long you train. This is the failure mode that looks like “LoRA does nothing” — loss curve flat at the pretrained value.
• Both random. Both gradients are nonzero and the pair trains, but you have given up the step-0 no-op property and injected noise at full weight scale. In PEFT this is reachable deliberately: init_lora_weights=False means “random init of A and B”, and the docs say it is intended for debugging. Use it to reproduce bugs, not to train.

The one-line answer. $B$ is zero so the adapter is exactly a no-op initially; $A$ is random so that this no-op is not also a gradient-free saddle.

2.3 Why the scale is $\alpha /r$ and not $\alpha$

Start from what $\Delta W$ actually is. $BA={\sum }_{j=1}^r{b}_j{a}_j^{\top }$ — a sum of $r$ rank-1 outer products, where $b_j$ and $a_j$ are columns/rows of the factors. The per-element magnitude of that sum therefore grows with how many terms you are adding. Hold $\alpha$ fixed, increase $r$, and the update gets larger for free: the effective learning rate on the layer has silently changed.

Dividing by $r$ cancels that growth, so the expected contribution per element is approximately independent of $r$. The practical consequence is the one that shows up in every tuning guide: $\alpha$ becomes a gain you set once, and sweeping $r$ no longer requires re-tuning the learning rate. Conventions in the wild — $\alpha =2r$, $\alpha =r$, or fixed $\alpha$ — are exactly that, conventions; the LoRA paper sets $\alpha$ once and scales, and there is no universal optimum.

This is also why $\alpha /r$ is not the same as “set the learning rate lower”. The scale multiplies the forward contribution of the adapter, so it also scales the gradient with respect to both $A$ and $B$ by the same factor. It is a reparameterisation of the adapter branch, not an optimizer setting.

rsLoRA: when $r$ is large, the exponent is contested. The $\alpha /r$ argument assumes the $r$ rank-1 terms add roughly coherently. If instead you assume they are independent and their contributions add in quadrature, the sum grows like $\sqrt{r}$, and the correct normaliser is $\alpha /\sqrt{r}$ — which is what Rank-Stabilized LoRA proposes (Kalajdzievski 2023). PEFT exposes it directly: use_rslora=True sets the scaling to lora_alpha/math.sqrt(r) instead of lora_alpha/r (PEFT LoraConfig docstring, as of 2026-10, PEFT 0.21.2). The rule of thumb that falls out: below $r\approx 64$ the two agree closely enough that it rarely matters; at high rank under-scaled adapters under-train and rsLoRA helps.

LoRA+ decouples the two factors’ learning rates. Because $\partial \mathcal{L}/\partial B$ scales with $A$ and $\partial \mathcal{L}/\partial A$ with $B$, the matrices are not on equal footing; Hayou et al. 2024 argues $B$ wants a larger rate — a ratio far above 1 (the figure usually quoted is 16×, [UNVERIFIED] here). Version sensitivity: loraplus_lr_ratio existed on older LoraConfig versions but is not in PEFT 0.21.2 [as of 2026-10]; today you implement it by splitting lora_A and lora_B into two optimizer groups.

DoRA is the other mainstream variant. Weight-Decomposed Low-Rank Adaptation splits $W$ into a per-column magnitude $m$ and a direction, trains $m$ directly and applies LoRA to the direction: $W^{\prime }=m(W_0+BA)/\Vert W_0+BA{\Vert }_c$, where $\Vert \cdot {\Vert }_c$ is the column-wise L2 norm (each column is rescaled to unit norm and the learned magnitude vector $m$ restores its length) (Liu et al. 2024). Its motivation is a measured difference in learning pattern: full fine-tuning shows a negative correlation between magnitude and direction change, LoRA a positive one, DoRA’s closer to full FT’s; it reports +3.7/+1.0 on LLaMA-7B/13B commonsense reasoning over LoRA. Enable with use_dora=True, at the cost of extra training memory and a bigger unmerged inference overhead.

2.4 How many parameters you are actually training

Per adapted matrix the trainable count is

$$\underset{\text{A}}{\underbrace{r{d}_{in}}}+\underset{\text{B}}{\underbrace{r{d}_{out}}}=r(d_{in}+d_{out}),\text{versus}{d}_{in}{d}_{out}\text{for the frozen weight.}$$

Work a real model: LLaMA-2 7B has $d_{model}=4096$, 32 layers, FFN width 11008; the attention projections are all $4096\times 4096$ and the MLP is $4096\times 11008$ / $11008\times 4096$.

The all-linear row in detail, since this is the arithmetic an interviewer wants to see: per layer, $4$ attention matrices $\times 16(4096+4096)=524,288$; gate_proj and up_proj $\times 16(4096+11008)=483,328$; down_proj $=241,664$; total $1,249,280$; times 32 layers $=39,976,960\approx 40$ M.

Two things that surprises people:

• The MLP dominates, not attention. The MLP holds roughly two thirds of a modern transformer’s parameters, so “adapt all linear” is mostly an MLP decision.
• The adapter is 80 MB, not 80 KB. 0.6% of the model is small but not nothing; the storage win versus a 14 GB full checkpoint is still ~170×.

The choice of which modules matters more than $r$ for quality. The original paper found q/v sufficed for its benchmarks; QLoRA found the opposite at 4-bit, reporting that “LoRA on all linear transformer block layers are required to match full finetuning performance”, and that $r$ itself “do[es] not affect performance” (Dettmers et al. 2023, §4).

2.5 QLoRA: 4-bit storage, 16-bit compute

QLoRA adds three things on top of LoRA, and it is worth being precise about what each buys because only one of them is a pure memory win.

flowchart TB
    subgraph Storage["4-bit storage"]
        W4["W_NF4 blocks (4 bits/param)"]
        C2["c2: fp8 scales, block 256"]
        C1["c1: fp32 scales, block 256"]
        C2 --> C1
    end
    W4 --> DQ["doubleDequant -> bf16"]
    C2 --> DQ
    C1 --> DQ
    X["x (bf16)"] --> MM["matmul in bf16"]
    DQ --> MM
    X --> LB["B(A x) in bf16"]
    LB --> ADD["Y = X W + (alpha/r) X L1 L2"]
    MM --> ADD

1. NF4 — 4-bit NormalFloat. The format is built from the quantiles of a standard normal, and §2.4 of the quantization chapter derives it. The one-sentence version: weights are assumed roughly zero-mean Gaussian, so instead of 16 evenly spaced levels you place levels at the Gaussian’s quantile midpoints, giving every level equal probability mass. The assumption is load-bearing and is the reason the failure mode in §5 exists.

2. Double quantisation. A 4-bit format needs a small block size for precision, and block scales are not free. With fp32 scales and block size 64 the scales alone cost $32/64=0.5$ bits per parameter. QLoRA quantises the scales themselves to 8-bit with a second-level block of 256, giving

$$\frac{8}{64}+\frac{32}{64\cdot 256}=0.125+0.00195\approx 0.127\text{bits/param},$$

a saving of 0.373 bits per parameter — about 3 GB on a 65B model. The paper’s ablation notes DQ “only leads to minor gains” in accuracy; its real job is fitting a model into the next GPU size down.

3. Paged optimizers. Gradient checkpointing saves memory but creates transient spikes — one long sequence can momentarily need far more memory than the steady state. Paged optimizers use NVIDIA unified memory to page the optimizer states between GPU and host RAM instead of OOM-ing (Dettmers et al. 2023, §3). The paper does not benchmark this well: paging only triggers on those spikes, and at batch size 16 it observed the same training speed as a regular optimizer.

The compute policy is the part people get wrong. QLoRA has one storage type (4-bit NF4) and one computation type (bf16). Every matmul dequantises the block on the fly to bf16 and multiplies in bf16; only the adapters and their optimizer states live in 16-bit. So QLoRA is 4-bit storage, 16-bit compute — it saves memory, not FLOPs, and step time is usually worse than 16-bit LoRA because of the dequantisation overhead. The paper’s own 7B breakdown shows why base quantisation is not the whole story: for LLaMA 7B the 4-bit base took 5,048 MB, LoRA input gradients 567 MB (18 MB per sequence with gradient checkpointing), and the LoRA parameters only 26 MB. Gradient checkpointing buys you more than shrinking the adapter does.

3. Implementation

Real code against peft==0.21.2 and transformers (October 2026). PEFT defaults worth knowing, because they differ from what most tutorials set explicitly: r=8, lora_alpha=8, lora_dropout=0.0, bias="none", use_rslora=False, use_dora=False (LoraConfig, as of 2026-10).

# peft==0.21.2, torch>=2.4
from peft import LoraConfig, TaskType, get_peft_model

config = LoraConfig(
    r=16,                      # rank of the correction; 8-64 is the useful range
    lora_alpha=32,             # alpha/r = 2.0 here; see §2.3
    lora_dropout=0.05,         # regularisation; 0.0 is PEFT's default
    bias="none",               # do NOT train biases unless you must
    task_type=TaskType.CAUSAL_LM,
    # all-linear covers q/k/v/o + gate/up/down and skips lm_head for PreTrainedModel
    target_modules="all-linear",
    init_lora_weights=True,    # True => B := 0, A := random  (the §2.2 no-op property)
    use_rslora=False,          # set True if you push r above ~64
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 39,976,960 || all params: 6,738,415,616 || trainable%: 0.5933

The same stack in 4-bit, which is QLoRA:

# bitsandbytes==0.50.2, transformers (2026), peft==0.21.2
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # transformers default is "fp4"; QLoRA uses NF4
    bnb_4bit_use_double_quant=True,         # transformers default is False
    bnb_4bit_compute_dtype=torch.bfloat16,  # transformers default is torch.float32
    bnb_4bit_quant_storage=torch.uint8,     # default; set bf16 if you shard with FSDP
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb, device_map="auto"
)
model = prepare_model_for_kbit_training(model, use_gradient_checkpointing=True)
model = get_peft_model(model, config)

Three of those lines are defaults you are overriding, and the defaults are wrong for QLoRA: bnb_4bit_quant_type defaults to "fp4" and bnb_4bit_compute_dtype to torch.float32 in BitsAndBytesConfig [as of 2026-10]. The QLoRA paper evaluates FP4 and finds it “consistently one percentage point behind” NF4 on 5-shot MMLU. Leaving compute_dtype at fp32 roughly doubles activation memory for no quality gain.

Merging, when you are done:

merged = model.merge_and_unload(safe_merge=True)   # NOT in-place; assign the result

The signature is merge_and_unload(progressbar=False, safe_merge=False, adapter_names=None) [as of 2026-10]. safe_merge=True checks for NaNs in the adapter weights before writing them into the base — cheap, and it turns a silently corrupted model into a loud error.

Knobs and their realistic ranges:

4. Numbers and trade-offs

Everything below is arithmetic from the byte counts in §1, for a 7B model ($d_{model}=4096$, 32 layers) at $r=16$ all-linear (40.0 M trainable). Activation memory is the one term that depends on batch and sequence length and is marked (example).

Derivation notes so the table is checkable:

• Full fine-tune weights/gradients $=2\times 7\times 10^9=14$ GB each; Adam $m$ and $v$ $=4\times 7\times 10^9=28$ GB each; master weights another 28 GB. Subtotal 112 GB — the 16 bytes/param of §1, and the same accounting as the SFT and pretraining chapters. Drop the fp32 master copy and you are at the 12 bytes/param / 84 GB convention the PPO chapter uses, which is why that chapter’s totals are lower.
• QLoRA base $=7\times 10^9\times (4+0.127)\text{bits}/8=3.61$ GB. The QLoRA paper measured 5,048 MB for its 4-bit LLaMA-7B, higher than this figure because the embeddings and lm_head are left in higher precision and because of kernel padding — budget ~5 GB, not 3.6 GB.
• Adapter optimiser state $=40\times 10^6\times 8$ B $=0.32$ GB, master weights $0.16$ GB — the optimizer state scaled down ~175×, because $40$M parameters is 0.6% of the $7$B full-fine-tune set ($8$ B/param $\times 7\times 10^9=56$ GB versus $8$ B/param $\times 40\times 10^6=0.32$ GB).

Which axis each choice moves:

Inference is a separate question. Merged, LoRA costs nothing — the model is exactly the same shape and size. Unmerged, the adapter adds $2r(d_{in}+d_{out})$ multiply-adds per token per layer, a few percent of the forward pass at $r=16$ all-linear. Serving many adapters over one base (a per-customer fine-tune, say) is its own engineering problem, solved by kernels that batch different adapters in the same forward pass — see Serving: vLLM and PagedAttention.

5. Pitfalls

Learning rate set to the full-fine-tuning value. The single most common false negative. Symptom: loss decreases slowly, plateaus clearly above the full-FT baseline, and everything looks like “LoRA is just worse”. Diagnosis: only 0.6% of parameters receive gradient, and those gradients are scaled by a random-init factor, so the effective step size is small. Fix: 1e-4 to 3e-4 for LoRA/QLoRA versus 1e-5 to 2e-5 for full FT, then sweep $\alpha$ together with it — the two are coupled through $\alpha /r$.

Rank bumped without re-checking $\alpha$. Symptom: going from $r=8$ to $r=64$ at fixed lora_alpha makes training unstable or the adapter dominate the base weights. Diagnosis: the update scale grew with $r$ (§2.3). Fix: keep $\alpha /r$ constant, or turn on use_rslora at high rank.

Expecting merge_and_unload() to be in-place. Symptom: you call it, then evaluate model, and the quality is unchanged. Diagnosis: it returns a new module; the original is untouched. merged = model.merge_and_unload() — always assign.

Numerical drift when merging. Symptom: merged-model eval differs slightly from adapter-on-base eval; usually small, occasionally large enough to flip task scores. Diagnosis: W_0 + (α/r)BA is computed in the dtype of the base weights — bf16 has ~3 decimal digits, so the merged matrix is a rounded version of the sum. The merge is mathematically exact, numerically not. Mitigation: safe_merge=True to catch NaNs, merge in fp32 when the base allows it, and treat eval-after-merge as a separate measurement (§7) rather than assuming it is identical.

Merging an adapter into a separately-quantised base. Symptom: the merged 4-bit model is clearly worse than running the un-merged 4-bit base plus adapter, and the gap is much bigger than the usual bf16 drift. Diagnosis: QLoRA quantised $W_0$ first and then trained $\Delta W$ against those quantised weights; the adapter is tuned for $\text{quant}(W_0)+\Delta W$. Writing it back as $\text{quant}(W_0+\Delta W)$ applies a second, unmodelled rounding. It is not the same function. Fix: merge into a 16-bit base and re-quantise, or serve base-plus-adapter unmerged. PEFT even warns about this class of issue for adapters whose initialisation mutates the base weights.

Quantising the wrong layers. Symptom: QLoRA loss is fine but generation quality is poor; or print_trainable_parameters() shows a suspiciously round 0 (no modules matched). Diagnosis A: target_modules names that do not exist in your architecture — e.g. using LLaMA’s q_proj/down_proj on a GPT-2-style model, whose projections are c_attn/ c_proj. Pass the wrong strings and PEFT replaces nothing. Fix: prefer target_modules="all-linear", which resolves per architecture. Diagnosis B: quantising lm_head and the embeddings. Fix: keep the output head and embeddings in high precision; bitsandbytes exposes llm_int8_skip_modules, and Transformers leaves lm_head alone by convention.

Training the biases “just in case”. Symptom: disabling the adapter no longer recovers the base model’s behaviour. Diagnosis: bias="all" or "lora_only" trains bias vectors outside the adapter, so they survive adapter removal — PEFT’s docstring warns about this. Fix: bias="none" unless you have a measured reason.

Adapting for new knowledge rather than new behaviour. Symptom: the model learns the format of your data but keeps hallucinating facts, or fails on a new script/language. Diagnosis: a rank-16 correction can rewire style and formatting; it cannot store a knowledge base. Fix: retrieval for facts, full fine-tuning for genuinely new capabilities. This is the subject of §6.

6. Using it in a project

Use LoRA when you are changing how the model responds (format, tone, tool-call schema, domain jargon, a classification head) and you have somewhere between a few thousand and a few hundred thousand examples. Use QLoRA when the same is true but the base does not fit in your memory, or when you want to iterate on many variants cheaply.

When NOT to reach for LoRA:

• Under ~1,000 examples. You do not have enough signal to move any parameters usefully. Prompt engineering with 5–20 well-chosen few-shot examples, or retrieval, will beat a LoRA trained on 800 rows — and it is free and instantly reversible. Come back to LoRA when you have a task you cannot prompt your way out of.
• Adding knowledge. Facts, new languages with new scripts, or a proprietary corpus are a poor fit for a low-rank update. Use RAG if the knowledge is retrievable; full fine-tuning if it must be in the weights.
• A small model that already fits comfortably. If your model is 1B parameters, full fine-tuning fits on one GPU, trains in minutes, and has no adapter/merge machinery to get wrong. LoRA’s advantage shrinks as the base shrinks.
• Latency-critical serving with no merge step. If your serving stack cannot merge adapters and cannot batch them, the unmerged adapter is a permanent per-token overhead. Merge before deploying, or use a multi-adapter server.
• Hardware without 4-bit kernels. QLoRA needs bitsandbytes’ CUDA kernels (or an equivalent). On CPUs and some accelerators 4-bit storage dequantises back to fp16 for the matmul, so you pay the memory saving but not the speed — see Quantization for the hardware table.

When NOT to quantise the base (i.e. plain LoRA instead of QLoRA): when the 16-bit model already fits. You are then paying a lossy merge path, dequantisation overhead and a small quality cost for nothing. The QLoRA paper’s own framing is that it matters “to fit models of certain size into certain GPUs”, not that it is better.

The decision in one pass:

1. Can prompting or retrieval solve it? → do that, stop.
2. Do you need new knowledge? → RAG, or full fine-tune.
3. Does the 16-bit model fit for training? → LoRA, target_modules="all-linear", $r$=16, LR 2e-4.
4. Otherwise → QLoRA, same config, NF4 + double quant, gradient checkpointing on.
5. Merge and re-evaluate before shipping. If the merge changes results, serve unmerged.

If you are choosing this path because you are about to do preference optimisation on top of it, note that the same adapter machinery is what RLHF: The Three Stages uses to keep a 7B policy model on one GPU.

7. How you would prove it worked

Metric and baseline. The baseline is the same base model with no adapter, and the question is whether the adapter beats it by more than run-to-run noise on a held-out set the adapter never saw. Report a task metric (exact-match on the output format, or accuracy on a labelled eval), not training loss. Training loss on LoRA always looks good, because you are fitting the adapter to a small slice of the model.

Four measurements, in increasing order of usefulness:

1. Held-out task metric, base vs adapter vs (if you can afford it) full fine-tune. This is the only number that answers the actual question.
2. Merge equivalence. Run the same eval with the adapter attached and with the adapter merged. (example) Expect max-abs logit differences around 1e-2 in bf16 and identical accuracy; a task-score gap means your merge path is broken.
3. Forgetting check. Run a general benchmark (an instruction-following or MMLU-style set) before and after. LoRA should lose much less than full FT here; a large drop means your learning rate or rank is too high.
4. Ablation on target_modules. q,v-only vs all-linear, same $r$ and data. This is the experiment that reproduces the QLoRA finding on your task, and it usually shows the largest delta of anything you can tune.

Reporting template (example — replace with your own numbers). “Fine-tuned Llama-3.1-8B with LoRA ($r$=16, $\alpha$=32, all-linear, 1e-4 cosine, 2 epochs on 12k pairs). Held-out exact-match 71.2% vs 48.9% for the base model with a 5-shot prompt. Merging changed accuracy by 0.1 pp (within seed noise); MMLU dropped 0.4 pp vs 3.1 pp for the full fine-tune we ran as a control. QLoRA reproduced the LoRA number to within 0.6 pp at 1/3 the memory.”

8. Interview drill

“Why is $B$ initialised to zero?” Strong answer: two reasons. Forward: with $B=0$, $\Delta W=BA=0$, so the adapted model is exactly the pretrained model at step 0 and training starts from a known-good point. Backward: $\partial \mathcal{L}/\partial B\propto g(Ax{)}^{\top }$ is nonzero, so $B$ starts learning immediately, and once $B\ne 0$, $A$ receives gradient too. Initialising both to zero instead puts the pair on a saddle where both gradients vanish forever. Common wrong answer: “so the adapter doesn’t affect the model at the start.” True but incomplete — it does not explain why $A$ is random, and it invites the follow-up “why not initialise both to zero?” which has a different answer.

“Why divide by $r$ and not just use $\alpha$?” Strong answer: $BA={\sum }_j{b}_j{a}_j^{\top }$ is a sum of $r$ terms, so its magnitude grows with $r$; dividing by $r$ makes the expected per-element update roughly rank-independent, which decouples the rank sweep from the learning-rate sweep. $\alpha$ then acts as a fixed gain. Follow-up: “is $r$ the right exponent?” → not necessarily; rsLoRA argues $\sqrt{r}$ under an independence assumption and empirically wins at high rank. PEFT exposes it as use_rslora.

“How many parameters does LoRA train on a 7B model?” Strong answer: $r(d_{in}+d_{out})$ per adapted matrix. With $r=16$, all-linear, on LLaMA-2 7B: $32\times 1,249,280\approx 40.0$ M, about 0.59% of 6.74B, or an 80 MB bf16 adapter. Attention-only at $r=8$ is 4.2 M / 0.06%. Common wrong answer: “a few thousand parameters” — conflating rank with parameter count, or forgetting that the count is summed over 7 matrices in each of 32 layers.

“What does QLoRA actually quantise, and what stays in 16-bit?” Strong answer: the frozen base weights are stored in 4-bit NF4 with double-quantised scales; the LoRA adapters, their gradients and the optimizer states are bf16; and every matmul dequantises blocks to bf16 and multiplies in bf16. So it is 4-bit storage, 16-bit compute — a memory optimisation, not a compute optimisation. Common wrong answer: “it trains in 4-bit.” The weights are never updated and are never multiplied in 4-bit; gradients do not flow to them at all.

“When is LoRA the wrong choice?” Strong answer: (a) fewer than ~1k examples — prompt or retrieve instead; (b) adding new knowledge — low rank cannot store a corpus, use RAG or full FT; (c) a small model that already fits in memory, where LoRA’s saving is not worth the machinery; (d) hardware or a serving stack without 4-bit kernels or merge support, where you pay the overhead without the benefit. Biderman et al. quantify (b)-adjacent effects: LoRA trails full fine-tuning on code and math while forgetting less. Common wrong answer: “LoRA is always within 1% of full fine-tuning.” It depends on the task; the honest statement is that it matches on many instruction/style tasks and lags on tasks needing genuinely new capability.

“You merged a QLoRA adapter into the 4-bit base and quality dropped. Why?” Strong answer: the adapter was trained against $\text{quant}(W_0)$; merging as $\text{quant}(W_0+\Delta W)$ introduces a second rounding the adapter never saw. Also, $W_0+\frac{\alpha }{r}BA$ is computed in the base’s dtype, so even a bf16 merge is a rounded sum, not an exact one. Merge into a 16-bit base and re-quantise, or serve unmerged. Follow-up: “how would you detect it?” → evaluate the same held-out set with the adapter attached and with it merged, and use merge_and_unload(safe_merge=True) to rule out NaNs.

“Your adapter trains but the loss plateaus above the baseline. First three things you check?” Strong answer: (1) learning rate — LoRA typically needs 1e-4 to 3e-4, an order of magnitude above full FT; (2) target_modules — confirm something was actually replaced (print_trainable_parameters() and model.peft_config), and prefer all-linear; (3) rank and $\alpha$ — raise $r$, keep $\alpha /r$ sane. Only after those would I look at the data. Common wrong answer: “increase the rank.” Rank is usually the last thing that matters; the QLoRA paper found $r$ had little effect while which modules were adapted mattered enormously.

Next: Quantization — this chapter used 4-bit as a memory trick to make adapters fit; the next one treats low-precision arithmetic as the subject, derives the affine dequantiser that NF4 is a special case of, and quantifies what each format costs in accuracy.