Supervised Fine-Tuning (SFT)

TL;DR — A pretrained model is a text completer. Ask it “How do I reverse a linked list?” and it may answer with more questions, because that is what the web does next. SFT fixes the format: you train the same next-token cross-entropy objective on (prompt, response) pairs rendered through the model’s own chat template, and you mask the prompt tokens out of the loss so the gradient only teaches the assistant’s turns. It changes style dramatically with very little data — LIMA used 1,000 examples (~750k tokens) and was competitive with RLHF-trained models on human preference — because almost all knowledge is already in the base model; SFT mostly decides which behaviour to surface. SFT is cheap (hours of GPU, not months), SFT does not add facts, and the most common failure is training on the prompt tokens.

1. Why this exists

Take a released base model and give it a real user prompt:

Explain how TCP congestion control works.

What I would like to know:

A base model continues that text however the web would: another bullet of questions, a completed form, a Reddit reply. It has the knowledge and no idea that it is being asked to answer. This is not a knowledge failure and bigger models do not fix it — it is a distribution mismatch between “web text” and “an assistant’s turn”.

The first production demonstration is InstructGPT (Ouyang et al. 2022, arXiv:2203.02155): supervised fine-tuning on ~13k labeler demonstrations, then a reward model and PPO. The paper’s own headline is that the 1.3B InstructGPT model was preferred by human labelers over the 175B base GPT-3 — a 100× parameter gap closed by roughly 13k examples. The task was never capacity; it was output distribution.

The second observation is why this works with so little data. Zhou et al. 2023 (LIMA, arXiv:2305.11206) fine-tuned a 65B LLaMA on 1,000 carefully curated prompt/response pairs — about 750,000 tokens in total — and reported it competitive with or preferred over RLHF-trained models. Their explanation is the superficial alignment hypothesis: nearly all capability is learned during pretraining, and SFT teaches the format and interaction style in which that capability is expressed. If that is right, data quality and diversity dominate data quantity; LIMA’s own comparison is against Alpaca, trained on 52,000 self-instruct examples (Wang et al. 2022, arXiv:2212.10560).

Concrete failure #1 — training on the prompt tokens. If the loss covers the whole rendered conversation, roughly half your gradient (for a balanced split) teaches the model to produce user turns. The symptom is unmistakable once you look for it: generations that restate the question, continue with a plausible follow-up question, or flip roles mid-answer.

Concrete failure #2 — template mismatch. The model was instruction-tuned with one exact serialisation of roles into tokens. Train with a different one, or forget the generation prompt at inference, and it behaves like a text completer again. Nothing crashes; quality silently collapses.

Where this matters for you. In an interview, “why does SFT change style so much with so little data?” and “why mask the prompt tokens?” separate someone who has read about fine-tuning from someone who has run it.

2. How it works

2.1 What SFT actually optimises

SFT is the same objective as pretraining — next-token cross-entropy — on a different distribution: rendered chat transcripts. The only structural novelty is the loss mask. For a rendered token sequence $x_1,\dots ,x_T$ and a mask $m_t$:

$$L(\theta )=-\frac{1}{{\sum }_{t=1}^T{m}_t}\sum _{t=1}^T{m}_t\log p_{\theta }\left(x_t\mid x_{<t}\right)$$

• $x_t$ — the token at position $t$: simultaneously context for later positions and the target here.
• $x_{<t}$ — the causal context, identical to pretraining (Pretraining Pipeline §2.7). The one-position shift still applies: the logits emitted at $t-1$ are scored against $x_t$.
• $p_{\theta }(\cdot \mid x_{<t})$ — the next-token distribution after softmax. SFT changes nothing about the architecture or the head.
• $m_t\in \{0,1\}$ — the loss mask, and the only real novelty. It is 0 on every token belonging to a user turn (and the role markers around it) and 1 on tokens the assistant should learn to produce. In practice it is not a separate tensor: labels are set to the sentinel $\text{-100}$ at masked positions and F.cross_entropy(..., ignore_index=-100) drops them.
• ${\sum }_{t=1}^T{m}_t$ — the normaliser, i.e. the number of supervised tokens. This is the term people get wrong. F.cross_entropy with ignore_index sums the per-token losses and divides by the count of non-ignored targets, which is exactly $\sum m_t$.
• $-\log$ and the minus — as in pretraining, turning a likelihood into a non-negative loss in nats. $\exp (L)$ is again perplexity, now per supervised token — a different quantity from pretraining perplexity, and not comparable across datasets with different prompt/response ratios.

Why the normaliser is $\sum m_t$, not $T$. The two differ by the prompt fraction, which varies per example. Dividing by $T$ means an example that is 90% prompt contributes a loss scaled down ~10× relative to a 50/50 example. Because the gradient carries the same factor, the effective learning rate on the tokens you care about becomes a function of how long your prompts happen to be: the LR schedule stops meaning anything and “2 epochs” means different amounts of learning on different datasets. Dividing by $\sum m_t$ makes every supervised token worth exactly one token of gradient.

What happens if you remove the mask ($m_t=1$ everywhere): the model is trained to predict user text. Two costs. Gradient is diverted — with a 50/50 token split, half the update teaches prompt modelling. Worse, the model learns the joint distribution of conversations, which is the wrong object at sampling time: you condition on a prompt and want the assistant continuation, not a continuation of the transcript as a genre. TRL’s assistant_only_loss exists to avoid exactly this, and it defaults to False for language-modelling datasets, so a default configuration trains on everything.

The nuance. Shi et al. 2024 (arXiv:2405.14394) deliberately put the loss back on the instruction part — “Instruction Modelling” — and found it helps when instructions are long relative to outputs and when the example count is small, plausibly by regularising against overfitting the small SFT set. Masking is the right default, not a law.

2.2 Chat templates: the serialisation is part of the model

A chat template is a deterministic function from a list of {role, content} messages to one string (then to token ids). It ships in the tokenizer. For Llama 3 (Meta’s published prompt format, as summarised in the PyTorch torchtune docs):

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

How does TCP congestion control work?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

TCP adapts its send rate to perceived congestion...<|eot_id|>

Three properties matter:

• The markers are real special tokens, not text. In Llama 3 <|begin_of_text|> is token id 128000 and <|eot_id|> is 128009: each is a single id with its own embedding row, and the tokenizer emits them for you. Contrast Llama 2, where [INST] and <<SYS>> are ordinary text the model had to learn to parse. That is why “just write the tags inline” is a bug — typed as text, <|eot_id|> tokenizes into several unrelated pieces.
• Roles cannot be spoofed by content. Because the markers are reserved tokens with no text spelling, a user message containing the literal characters <|eot_id|> cannot inject a role boundary.
• The template is a trained interface. At inference you must apply the same template and truncate it at the generation prompt (add_generation_prompt=True), which emits the assistant header for the model to continue.

flowchart LR
    M["messages: system, user, assistant"] --> CT["tokenizer.apply_chat_template"]
    CT --> R["rendered string with special tokens"]
    R --> TK["token ids"]
    TK --> MK["build labels: prompt -> -100, assistant -> ids"]
    MK --> PK["optional packing into max_length blocks"]
    PK --> CO["collator: position_ids reset, cu_seqlens for FA varlen"]
    CO --> LS["masked cross-entropy"]
    LS --> UP["update weights"]

2.3 Where the SFT data comes from

Two dataset shapes: prompt-completion ({"prompt", "completion"}, where TRL infers completion_only_loss=True when the flag is left at None) and conversational / language-modelling ({"messages": [...]}, where the mask comes from the template and leaving the flag off means the loss covers the whole conversation; assistant_only_loss=True needs a template containing {% generation %} markers).

The practical rule from LIMA and from practitioner reports since: 1,000 diverse, correct, well-formatted examples beat 50,000 templated ones — and one inconsistently formatted subset (say, some responses wrapped in markdown fences and others not) teaches permanent inconsistency.

2.4 Packing, and its hazard

SFT data is heavy-tailed in length. If the mean example is 300 tokens and you pad every batch to max_length = 2048, you waste roughly 85% of each forward pass on padding. Packing concatenates several examples into one max_length block.

The hazard: a plain concatenation lets token i of document A attend to token j of document B. The model sees a context that never occurs at inference and learns spurious cross-document dependencies. The symptom is subtle — plausible but off-topic continuations, answers referencing a “previous conversation”, and a training loss noticeably below eval. Three things must ship together:

1. Reset position ids at each boundary, so RoPE (Positional Encoding) sees every document starting at position 0.
2. Mask attention between documents — a block-diagonal mask.
3. Tell the kernel the boundaries, i.e. cumulative sequence lengths for a varlen FlashAttention kernel. TRL’s padding-free collator emits position_ids, cu_seq_lens_q, cu_seq_lens_k, max_length_q, max_length_k and seq_idx, computed on CPU so the model does not derive them per layer.

Packing also interacts with truncation. Ding et al. 2024 (arXiv:2404.10830) show the naive “concatenate then cut at max_length” approach truncates many documents mid-sentence — the curse of truncation — producing models that hallucinate about code (a cut variable definition) and summarise unfaithfully. Best-fit packing truncates only when no legal grouping exists; the paper reports gains including +16.8% on a context-following measure. TRL exposes the same idea via packing_strategy: "bfd" (best-fit decreasing, truncates overflow), "bfd_split" (splits overflow so nothing is lost), "wrapped" (aggressive, cuts mid-sequence).

2.5 Why SFT, not RL, is the tool for format

Chu et al. 2025 (arXiv:2501.17161) compare SFT and RL head-to-head on rule- and vision-based variants of the same tasks: SFT memorises the training distribution and generalises poorly to out-of-distribution variants, while RL with an outcome-based reward generalises — and SFT is still necessary, because it stabilises the output format that RL then improves. That is SFT’s job: the format layer. If the model knows the task but produces it in the wrong shape, SFT fixes it. If it cannot reason about the task, SFT will memorise your examples and a verifier or RL will help.

3. Implementation

3.1 The version you should actually ship: TRL

trl==1.15.0 with transformers==5.19.0 [as of 2026-10]:

from datasets import load_dataset
from trl import SFTConfig, SFTTrainer

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    train_dataset=load_dataset("trl-lib/Capybara", split="train"),
    args=SFTConfig(
        output_dir="sft-model",
        max_length=2048,
        packing=True,               # packing_strategy defaults to "bfd" (best-fit decreasing)
        assistant_only_loss=True,   # mask everything except assistant turns
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,
        learning_rate=2e-5,         # SFTConfig default; raise for LoRA (see lora-qlora.md)
        num_train_epochs=2,
        bf16=True,
        logging_steps=10,
    ),
)
trainer.train()

Four facts here are not obvious:

• assistant_only_loss=True requires a template with {% generation %} markers and transformers>=5.18.0. TRL swaps in a training template when the model’s lacks them, and warns if the template’s stop token is not trained — masking the stop token means the model never learns to stop.
• packing=True with the default "bfd" strategy turns on padding_free automatically, and padding-free requires FlashAttention 2 or 3 for the varlen kernel.
• SFTConfig overrides several TrainingArguments defaults: learning_rate=2e-5, bf16=True, gradient_checkpointing=True, logging_steps=10.
• max_length defaults to 1024 with truncation_mode="keep_start", i.e. it keeps the beginning — which for a long prompt and a short answer cuts the answer off entirely. Set it from your data’s p99.

3.2 The mask, written by hand

transformers>=4.45; apply_chat_template(..., tokenize=True) returns a list of ids:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")

def encode_pair(prompt: str, completion: str, max_len: int = 2048):
    """Return (input_ids, labels) with the prompt masked to -100."""
    prompt_ids = tok.apply_chat_template(
        [{"role": "user", "content": prompt}],
        add_generation_prompt=True,   # emits the assistant header to continue from
        tokenize=True, return_dict=False,
    )
    full_ids = tok.apply_chat_template(
        [{"role": "user", "content": prompt},
         {"role": "assistant", "content": completion}],
        tokenize=True, return_dict=False,
    )
    # The completion starts where the two renders stop agreeing. They can diverge
    # BEFORE the end of prompt_ids, because the boundary can merge tokens or the
    # template can render a lone prompt differently. TRL computes this same
    # "common prefix length" and warns when it is shorter than the prompt.
    n = 0
    while n < min(len(prompt_ids), len(full_ids)) and prompt_ids[n] == full_ids[n]:
        n += 1

    labels = [-100] * n + full_ids[n:]     # -100 is ignored by cross_entropy
    return full_ids[:max_len], labels[:max_len]

And the loss, which must shift labels by one position exactly as in pretraining:

import torch.nn.functional as F

def sft_loss(model, input_ids, labels, attention_mask=None):
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits
    logits = logits[:, :-1, :]        # predict token t from position t-1
    labels = labels[:, 1:]
    # cross_entropy divides by the count of labels != ignore_index, i.e. by
    # sum(m_t) -- not by T. That is the masking denominator from section 2.1.
    return F.cross_entropy(logits.reshape(-1, logits.size(-1)),
                           labels.reshape(-1), ignore_index=-100)

The distinction to internalise: encode_pair decides which tokens carry loss; sft_loss must not re-introduce the ones you masked. Dropping the [:, :-1] / [:, 1:] shift is the same silent bug as in pretraining — loss goes to zero and generation is garbage.

3.3 The knobs that matter

4. Numbers and trade-offs

4.1 SFT is cheap; the data is the cost

For $N=65\times 10^9$ parameters, $D=750\times 10^3$ supervised tokens, 15 epochs, at MFU 0.40 on H100s (989 TFLOPS dense BF16), using $C\approx 6ND$:

$$C=6\cdot 65\times 10^9\cdot 750\times 10^3\cdot 15\approx 4.4\times 10^{18}\text{FLOPs}\frac{C}{0.40\cdot 9.89\times 10^{14}\cdot 3600}\approx 3\text{H100-hours (example)}$$

Compare the 8B/15T pretraining run in Pretraining Pipeline §4.1 at ~505,000 H100-hours. LIMA-style SFT is five orders of magnitude cheaper than pretraining. What you pay for is the human curation of 1,000 examples, not the GPU time — which is why “just fine-tune it” is a tempting but bad default: the compute is trivial, and the data and evaluation are the whole project.

4.2 Memory

Full fine-tuning needs ~16 bytes/parameter of optimizer state (2 bf16 weights + 2 bf16 grads + 4 fp32 master + 4 + 4 Adam moments), i.e. ~112GB for a 7B model before activations. ZeRO/FSDP (Rajbhandari et al. 2020, arXiv:1910.02054) shards that state across GPUs, and QLoRA (see LoRA and QLoRA) drops the frozen base to 4-bit and trains small adapters, which puts a 7B fine-tune on one 24GB consumer GPU.

4.3 How much of the sequence carries loss

Let $r=\sum m_t/T$ be the supervised fraction. Masking multiplies the number of supervised tokens per step by $r$, so reaching a given supervised-token budget needs roughly $1/r$ times as many steps. For balanced chat data $r\approx 0.4\text{–}0.6$; for long-document summarisation with a 4,000-token input and a 100-token summary, $r\approx 0.02$. Two consequences: compare runs in supervised tokens, not steps, and remember that val loss is per supervised token, so it is not comparable between configurations that changed the mask — ablate masking on generations, not loss curves.

4.4 Epochs: the counterintuitive result

InstructGPT states the finding plainly (Ouyang et al. 2022, §3.5): the SFT models overfit on validation loss after 1 epoch, and yet training for more epochs improved both the reward-model score and human preference ratings. LIMA trained for 15 epochs, selected checkpoints between epochs 5 and 10 using a 50-example development set, and noted explicitly that perplexity did not correlate with generation quality. The practical translation: do not early-stop SFT on validation loss.

5. Pitfalls

Training on the prompt tokens. Symptom: the model restates the question before answering, emits follow-up user questions, or flips roles mid-generation. The default for conversational data in most frameworks is to train on everything; you must ask for the mask.

Wrong or missing chat template. Symptom: the model completes your prompt as if it were a document — “What I would like to know:” gets more bullet points. Nothing errors; only generations reveal it. Always render through tokenizer.apply_chat_template and print one decoded example before launching.

Forgetting add_generation_prompt=True at inference. Symptom: the model generates the next role’s header (<|start_header_id|>user) instead of an answer, because it is continuing the raw transcript.

Masking the stop token. Symptom: generation never terminates and runs to max_tokens, producing a correct answer followed by invented turns. If <|eot_id|> / EOS is inside the masked region, the model is never reinforced for stopping; TRL warns about exactly this. The mirror image is forgetting to train the stop token because you truncated the sequence before it.

Packing without boundary handling. Symptom: training loss noticeably below eval, and plausible-but-off-topic continuations referencing a “previous conversation” that was another training example. Needs position-id reset plus block-diagonal attention (§2.4).

Truncation that cuts the answer. Symptom: the model emits truncated responses — a summary that stops mid-sentence, code with an unclosed block. With truncation_mode="keep_start" (the default) an over-long example keeps the prompt and loses the answer.

Early-stopping on validation loss. Symptom: the best-quality checkpoint is discarded, because val loss overfits at one epoch while preference keeps improving (§4.4).

Expecting SFT to add knowledge. Symptom: fluent, correctly formatted hallucination — the shape of an answer to a question whose facts the model does not have. Use RAG for facts. Gudibande et al. 2023 (arXiv:2305.15717) show imitation-tuned models match a stronger model’s style while trailing on capability.

Special tokens added without resizing embeddings. Symptom: a shape error on the embedding lookup, or a silently random embedding for the new token when an optimizer state is loaded afterwards. Call model.resize_token_embeddings(len(tokenizer)) and initialise those rows deliberately.

Double BOS. Symptom: <|begin_of_text|> appears twice; generations are subtly worse and may repeat their first sentence. The template already emits special tokens, so tokenize with add_special_tokens=False where the framework does not do it for you.

6. Using it in a project

Start by deciding not to fine-tune. The ladder, cheapest first — you should be able to say which rung you are on:

Fine-tune when all of these hold: the failure is format/style/behaviour rather than missing knowledge; prompting and few-shot have been tried and are unreliable in production; you have at least a few hundred high-quality examples and a way to make more when the distribution shifts; you can define a pass/fail metric on a held-out set (§7); and you have a reason to prefer weights over a prompt — lower latency, a smaller served model, or a behaviour prompts cannot reliably elicit.

Do not fine-tune when:

• You have fewer than ~100 examples. Few-shot prompting with a handful of excellent examples beats a 50-example fine-tune, and you keep the ability to change behaviour without retraining.
• The problem is knowledge. SFT will not insert facts; you get confident, well-formatted fabrication. That is a RAG problem, and the diagnostic is the gold-context test.
• The data changes weekly. Every change means a retrain. Retrieve instead.
• You cannot measure it. Without a held-out evaluation you cannot distinguish a successful fine-tune from a memorisation run.
• The real error is retrieval. Most “the LLM hallucinated” reports are retrieval failures; fix recall before touching weights.
• You are chasing capability, not format. If the base model cannot do the task at all, 1,000 examples teach it to imitate the shape of an answer without the substance.

Middle path: with 1k–50k examples and one GPU, use LoRA or QLoRA rather than full fine-tuning — most of the behaviour change, base left intact, several adapters over one served base.

7. How you would prove it worked

Metric. One primary number plus regressions. Primary: success rate on a held-out prompt set drawn from the same distribution as training but disjoint from it (200–500 items), graded by exact match, a rubric, or a judge; with production traffic, hold out a time slice rather than a random sample. Format adherence: the fraction of outputs that parse against your schema — the metric SFT should move most, and the cheapest to compute. Regressions: MMLU or a small general suite plus a refusal spot-check, because narrow fine-tuning is a classic way to silently lose general instruction following. Memorisation: prompt the model with training-example prefixes and measure verbatim completion; high held-out success and high memorisation means overfitting.

Baselines — all of them, or the number means nothing: base zero-shot, base few-shot with your best examples in the prompt, and the prompt-engineered production system if one exists.

Ablations that answer real questions: masked vs unmasked (compare generations, never loss — the denominator changed); packing on/off at identical supervised-token budget, with and without boundary masking; an epoch sweep with a quality metric; and a template swap to quantify how much the interface is baked in.

Reporting template (example — replace with your own numbers). “We fine-tuned an 8B instruct model on 4,200 (prompt, response) pairs for 3 epochs with assistant-only loss and best-fit packing. On a 400-item held-out set, exact match rose from 41% (base, zero-shot) and 55% (base, 5-shot) to 78% (example). MMLU moved from 68.1 to 67.6 (example), within noise. A no-mask ablation reached the same held-out loss in 60% of the steps but produced 9% role-flip generations (example). A packing-without-boundary-masking ablation showed a 0.15-nat train/eval gap (example) and off-topic continuations in 6% of samples (example).”

For judge-based evaluation, mind the biases in RAG Evaluation §2.3 — position, verbosity, self-preference. Swap order, control length, and judge with a different model than the generator.

8. Interview drill

“Why does SFT change behaviour so much with only a thousand examples?” Strong answer: the superficial alignment hypothesis — capability comes from pretraining and SFT selects the format and style in which it is expressed. LIMA showed 1,000 curated pairs (~750k tokens) competitive with RLHF models in human preference, beating Alpaca’s 52,000 examples. Quality and diversity dominate quantity. Common wrong answer: “1,000 examples inject a new skill.” They inject a new response distribution; if the base model cannot do the task, SFT teaches imitation of the answer’s shape.

“Why mask the prompt tokens in the loss?” Strong answer: you condition on the prompt and want the assistant distribution, so gradient spent on prompt tokens teaches the wrong object; and the mask is what makes the normaliser $\sum m_t$ instead of $T$, keeping the effective learning rate independent of prompt length. Follow-up: “always?” → no — Shi et al. 2024 find training on the instruction part can help with long instructions or few examples, plausibly by regularising overfitting. Common wrong answer: “masking makes training faster.” It makes each step cheaper in scope but you need more steps for the same supervised-token budget; it is a correctness choice.

“What is a chat template, and why are the role markers special tokens?” Strong answer: a deterministic renderer from messages to tokens; the markers are reserved single-token ids (Llama 3: <|begin_of_text|> is 128000, <|eot_id|> is 128009), which makes them unambiguous and un-spoofable by user content and lets the model attend to them as one unit. Llama 2’s [INST] was plain text by contrast. Common wrong answer: “a prompt prefix I write in the system message” — typed tags tokenize as ordinary text.

“How many epochs should SFT run?” Strong answer: it depends on dataset size — 1–3 for >100k examples, 3–15 for a few thousand curated ones. LIMA trained 15 and picked checkpoints from epochs 5–10; InstructGPT trained 16 and reported that validation loss overfits after one epoch while human preference keeps improving, so loss is the wrong selection rule. Common wrong answer: “train until the loss converges.”

“Does SFT teach the model new facts?” Strong answer: no — it teaches form. Facts need continued pretraining or RAG; low-rank adapters are especially poor at new knowledge. Gudibande et al. show imitation-tuned models match style while trailing on capability, and the symptom is confident, well-formatted hallucination. Common wrong answer: “yes, that’s how you make a model a domain expert” — the auditable route to domain facts is retrieval.

“You enable packing and the model starts producing off-topic continuations. What happened?” Strong answer: concatenated documents attend across boundaries. Fix with position-id reset per boundary plus a block-diagonal mask, or a varlen FlashAttention kernel fed cumulative sequence lengths; TRL’s padding-free collator emits position_ids and cu_seq_lens_* for exactly this. Common wrong answer: “packing is fine, it’s just concatenation” — attention does not stop at a token boundary unless you tell it to.

“SFT vs RLHF/DPO — which do you reach for?” Strong answer: SFT for format and behaviour imitation; preference optimisation when you can compare outputs pairwise or score them with a verifier, which is where generalisation to unseen variants comes from. Chu et al. 2025 show SFT memorises and RL generalises, and that SFT is still required to stabilise the format RL then improves — hence the standard pipeline SFT → preference optimisation. Common wrong answer: “RLHF replaces SFT.” It builds on the SFT checkpoint; see Part 03.

Next: LoRA and QLoRA — you now know the objective and the masking; the next question is how many parameters you actually need to update to get the behaviour change.