The Transformer

TL;DR — A Transformer is a stack of identical blocks, each of which is just two things: a mixing step (attention, which moves information between positions) and a computing step (a position-wise MLP, which processes each position independently). Around every one of those sits a residual connection, so the block’s input is added to its output and the stream flowing up the stack is never overwritten — only read from and added to. That residual stream is the mental model that makes the rest of the architecture obvious: norm layers keep its scale sane, the pre-norm placement keeps its gradient path clean, and depth is what lets the model compose operations. Attention is the only part that is $O(n^2)$; the MLP holds most of the parameters and most of the FLOPs at realistic context windows.

1. Why this exists

The 2017 paper’s own framing is the right one: this was not a better layer, it was a better graph.

A recurrent network computes $h_t=f(h_{t-1},x_t)$. Training it on a 1000-token sequence means 1000 dependent steps; the GPU’s parallel width is useless along the time axis, and the gradient from token 1000 to token 1 must survive 999 multiplies. A convolutional stack removes the sequential dependency but pays in path length: to relate two positions far apart you need $O(n/k)$ layers of contiguous kernels, or $O({\log }_{k}n)$ with dilations. Both designs force you to choose between parallelism and reach.

The Transformer’s claim in Table 1 of the paper is narrower and more useful than “it’s better”: self-attention is $O(n^2\cdot d)$ per layer with $O(1)$ sequential operations and $O(1)$ maximum path length between any two positions. You pay a quadratic bill in exchange for removing the architectural barrier to parallelism and long-range mixing. Everything else — the hardware that got built, the scaling laws that fit, the fact that GPT-4-class models are decoder-only Transformer stacks — follows from that trade.

Here is the concrete failure the original design fixes. Take a task where the answer depends on two tokens 800 apart, e.g. “the error code in section 3 is which one, given the fix described in section 40?” An LSTM must route that dependency through 800 cell-state updates; in practice the signal is gone and the model attends to recency instead. Symptom in the product: the model answers using the last thing it read rather than the right thing, and retrieval quality is blamed when the architecture is the culprit.

What this chapter is and is not. Attention derives the mixing operation itself. This chapter is the block that wraps it, the stream that carries it, and the reason a 32-deep stack trains at all. If you can draw the block from memory and justify every arrow, this chapter did its job.

2. How it works

2.1 The one idea: the residual stream

Write ${\mathbf{x}}_i^{(\ell )}\in {\mathbb{R}}^d$ for the vector at position $i$ after layer $\ell$. Every modern block has the shape

$${\mathbf{x}}^{(\ell +1)}={\mathbf{x}}^{(\ell )}+\mathrm{Sublayer}(\mathrm{Norm}({\mathbf{x}}^{(\ell )}))$$

and the whole point is the leading ${\mathbf{x}}^{(\ell )}$.

• ${\mathbf{x}}^{(\ell )}$ — the residual stream. It is a $d$-dimensional vector per position that is created once by the embedding and then never replaced. Every sublayer reads from it and writes an increment back into it. Without this identity path, each block would be a function that destroys its input: the only way information from layer 1 could reach layer 32 would be through 31 nonlinear transforms, and the gradient would have to survive the same journey backwards.
• $\mathrm{Sublayer}(\cdot )$ — the increment: either the attention output or the MLP output. It is not the state; it is a correction to the state. Because the residual is additive, sublayers compose by summation, which is what makes the “write to a shared bus” interpretation (Elhage et al., A Mathematical Framework for Transformer Circuits, 2021, https://transformer-circuits.pub/2021/framework/index.html) legitimate rather than merely poetic.
• The absence of a scale on the residual — no $\alpha {\mathbf{x}}^{(\ell )}$ with $\alpha <1$ — is deliberate. Any per-layer decay would bound the effective path length between layer 1 and layer $L$, reintroducing exactly the vanishing-gradient problem the design removed.

If you delete the residual connection, a 12-layer Transformer still trains but noticeably worse and becomes far more sensitive to learning rate; at 32 layers it typically does not converge. It is the cheapest ablation in the field.

2.2 The block, end to end

flowchart TB
    X["x (n × d)"] --> R1["+"]
    X --> N1["Norm"]
    N1 --> ATT["Self-attention<br/>(mixes across positions)"]
    ATT --> R1
    R1 --> R2["+"]
    R1 --> N2["Norm"]
    N2 --> MLP["Feed-forward<br/>(each position alone)"]
    MLP --> R2
    R2 --> OUT["x (n × d) → next block"]

    style R1 fill:#2E7A5A,stroke:#1E6A4A,color:#fff
    style R2 fill:#2E7A5A,stroke:#1E6A4A,color:#fff

Formally, with $n$ the sequence length and $d$ the model width:

$$Z=X+\mathrm{Attention}(\mathrm{Norm}(X)),X^{\prime }=Z+\mathrm{MLP}(\mathrm{Norm}(Z))$$

• $X\in {\mathbb{R}}^{n\times d}$ — the current block’s input: one row per position, all rows carried on the same residual stream.
• $\mathrm{Attention}(\cdot )$ — the only operation in the block that lets row $i$ see row $j$. Its internals are the subject of Attention; here it matters only that it is position-aware (via positional encoding) and sequence-mixing.
• $\mathrm{MLP}(\cdot )$ — applied to each row independently and with the same weights, so it is a per-position transform. It cannot move information between positions; it can only reshape what is already there.
• Two norms, not one, and one per sublayer. The norm belongs to the sublayer it feeds, not to the block.

The sequencing is the load-bearing part: the block alternates mix → think. Attention lets positions average each other; the MLP then applies a learned nonlinearity to the result. Neither is sufficient alone — attention with no MLP can only re-weight existing embeddings, and an MLP with no attention is a per-token function, no better than a bag of words for anything order-dependent.

2.3 The feed-forward network is not “just an MLP”

$$\mathrm{MLP}(x)=W_2\sigma (W_{1}x+b_1)+b_2$$

• $W_1\in {\mathbb{R}}^{d_{ff}\times d}$ — the up-projection. It lifts the residual stream into a wider space ($d_{ff}=2048$ against $d=512$ in the 2017 base model; $4d$ is the classic ratio). The expansion is what creates room for the activation to act per-feature; without it, the activation is applied in the same basis as the residual reads and the layer has far less representational room.
• $\sigma$ — the elementwise nonlinearity. The 2017 paper uses $\max (0,\cdot )$ (ReLU); modern models largely use gated variants (SwiGLU, Shazeer 2020, https://arxiv.org/abs/2002.05202). Without some nonlinearity the two linears collapse into one and the block becomes linear — attention included, since the whole thing is then a matrix product.
• $W_2\in {\mathbb{R}}^{d\times d_{ff}}$ — the down-projection back to the residual width, so the increment can be added to the stream.
• $b_1,b_2$ — biases. The 2017 paper has them; most modern LLMs drop them from the MLP and attention projections (LLaMA, PaLM, Mistral, Qwen). The reason is partly that a norm sits on the sublayer’s input, so a constant shift can be partly absorbed by $\beta$ of the preceding norm, and partly that removing them saves parameters and a kernel launch for no measured loss.

Two numbers make the MLP matter. At $d_{ff}=4d$ it holds $8d^2$ parameters while the attention projections hold $4d^2$ — so $8d^2$ of the block’s $12d^2$ total, or two-thirds of every block’s weights, live in the MLP. That is why “the model’s knowledge is in the MLP” is a reasonable first guess when you are deciding what to freeze, quantize, or ablate.

2.4 Pre-norm vs post-norm: where the norm goes

There are two placements, and the difference is not cosmetic.

Post-norm (the 2017 paper). $$x^{\prime }=\mathrm{Norm}(x+\mathrm{Sublayer}(x))$$

Here the norm is on the residual path: the identity ${\mathbf{x}}^{(\ell )}$ you just added gets renormalised before it continues. Xiong et al. 2020 (On Layer Normalization in the Transformer Architecture, ICML 2020, https://arxiv.org/abs/2002.04745) prove with mean-field analysis that at initialization the expected gradients of parameters near the output layer of a post-norm stack are large, and that this is precisely why a learning-rate warmup is mandatory. The practical consequences:

• You need a warmup schedule or the run diverges.
• The required warmup grows with depth, so scaling the stack means retuning optimization, not just the model.

Pre-norm (most modern LLMs). $$x^{\prime }=x+\mathrm{Sublayer}(\mathrm{Norm}(x))$$

Now the norm is inside the increment, and the residual path is a pure identity. Xiong et al. show the gradients are well-behaved at initialization and the warmup can be dropped. That is why the pre-2020 encoder-decoder recipes treated warmup as mandatory and why modern recipes treat it as an optimization detail: warmup is still almost always used — it rarely hurts — but a pre-norm run without it no longer diverges.

The cost of pre-norm: the residual stream’s norm grows monotonically with depth (each block adds an increment), so the final norm before the output head is doing real work — it is not decoration. A side effect worth knowing: because the stream grows, per-layer increments become relatively smaller as you go deeper, which is one motivation for scaled residual-init schemes.

2.5 Layer norm: what each symbol does

$$\mathrm{LN}(x)=\gamma \odot \frac{x-\mu }{\sqrt{{\sigma }^2+ϵ}}+\beta$$

• $x\in {\mathbb{R}}^d$ — one position’s vector, on the residual stream. The norm is applied per position, normalizing across the $d$ features; it never mixes across the sequence dimension.
• $\mu =\frac{1}{d}{\sum }_j{x}_j$ — the mean across features. Subtracting it re-centers the vector at the origin. Removing this term gives RMSNorm (below).
• ${\sigma }^2=\frac{1}{d}{\sum }_j(x_j-\mu {)}^2$ — the variance across features. Dividing by $\sqrt{{\sigma }^2+ϵ}$ rescales the vector to unit scale. This is the term that actually keeps training stable: it makes the sublayer’s input statistics independent of how large the residual stream has grown.
• $ϵ$ — a small constant (commonly $10^{-5}$ to $10^{-6}$) that keeps the denominator away from zero when the vector is (near-)constant. Without it, a saturated or collapsed activation produces a division by ~0 and NaNs.
• $\gamma \in {\mathbb{R}}^d$ — the learned scale. It restores the model’s ability to represent features of different magnitudes after normalization; without it, every sublayer would be forced to read unit-variance inputs.
• $\beta \in {\mathbb{R}}^d$ — the learned shift. It restores a per-feature offset. This is the parameter RMSNorm drops, on the empirical finding (Zhang & Sennrich 2019, https://arxiv.org/abs/1910.07467) that the re-centering is not what is doing the work — the re-scaling is. RMSNorm is therefore cheaper and has fewer parameters, and it is what LLaMA, PaLM, Gemma, Mistral, Qwen and DeepSeek use.

The interviewer’s version of this question. “Why layer norm and not batch norm?” Strong answer: batch norm computes statistics across the batch, so at inference you must freeze running estimates, and the statistics depend on sequence padding and on batch composition — fatal for variable-length text and for the autoregressive decode loop, where batch size is 1. Layer norm normalizes each position using only that position’s own features, so train and inference behave identically and batching does not change the result.

2.6 Encoder, decoder, and cross-attention

The original architecture is a stack of $N=6$ identical encoder layers and a stack of $N=6$ identical decoder layers, plus one extra sublayer in the decoder. The three attention layouts are the whole difference:

• Cross-attention’s $Q$ has $n_{dec}$ rows while $K,V$ have $n_{enc}$ rows. The score matrix is therefore $n_{dec}\times n_{enc}$ — rectangular, which is fine, and is the reason a seq2seq model can read a source of a different length than its target.
• There is no causal mask on the encoder axis because the whole source is available at once; masking it would hide input the decoder is entitled to.
• Encoder self-attention is bidirectional: masking it costs you exactly the right-context signal that makes BERT embeddings good for retrieval.

Decoder-only models (GPT and everything since) keep only the causal self-attention column. BERT vs GPT vs Encoder-Decoder does the objective-by-objective comparison; Modern Architecture Deltas covers what changed in the block since 2017 (pre-norm, RMSNorm, SwiGLU, GQA, FlashAttention, MoE).

2.7 Two 2017 details: the $\sqrt{d}$ scale and weight tying

The paper multiplies the token embedding by $\sqrt{d_{model}}$ before adding the positional encoding. With a norm immediately downstream, the pre-norm value does not affect forward outputs at all — the norm divides it out — and it matters only for initialization calibration: the embedding rows start with small variance, and $\sqrt{d}$ puts them in the regime the norm’s $\gamma$ (initialized to 1) expects. Delete it and the model still trains; the symptom is a slightly worse early loss curve, not a break.

Weight tying is the second: the input embedding, the output embedding, and the pre-softmax projection are the same matrix. That is $d\times V$ parameters saved (16.4M of the base model’s ~60M, 27%), and it encodes the prior that a token’s input representation and its output logit direction should be related.

3. Implementation

Runnable block and stack, torch==2.4.0 (also correct on 2.1+), CPU-only:

import math
import torch
import torch.nn as nn
import torch.nn.functional as F


class TransformerBlock(nn.Module):
    """Pre-norm block: x = x + attn(norm(x)); x = x + mlp(norm(x))."""

    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.0):
        super().__init__()
        assert d_model % n_heads == 0
        self.n_heads, self.d_k = n_heads, d_model // n_heads

        # QKV in one matmul: 3x fewer kernel launches than three Linears.
        self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
        self.out_proj = nn.Linear(d_model, d_model, bias=False)
        self.norm1 = nn.LayerNorm(d_model)   # owns the attention sublayer
        self.norm2 = nn.LayerNorm(d_model)   # owns the MLP sublayer
        self.mlp = nn.Sequential(
            nn.Linear(d_model, d_ff, bias=False),
            nn.GELU(),
            nn.Linear(d_ff, d_model, bias=False),
        )
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, causal: bool = True):
        B, T, _ = x.shape

        # --- sublayer 1: attention, operating on the NORMED copy ---
        h = self.norm1(x)
        q, k, v = self.qkv(h).view(B, T, 3, self.n_heads, self.d_k) \
                          .permute(2, 0, 3, 1, 4)          # each (B, H, T, d_k)
        attn = F.scaled_dot_product_attention(q, k, v, is_causal=causal)
        attn = attn.transpose(1, 2).reshape(B, T, -1)
        x = x + self.dropout(self.out_proj(attn))      # <-- the residual

        # --- sublayer 2: MLP, position-wise, again on the NORMED copy ---
        x = x + self.dropout(self.mlp(self.norm2(x)))  # <-- the residual
        return x


class TinyTransformer(nn.Module):
    """Decoder-only stack, pre-norm, final norm before the head."""

    def __init__(self, vocab: int, d_model: int, n_heads: int, d_ff: int,
                 n_layers: int, max_len: int = 512):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab, d_model)
        self.pos_emb = nn.Embedding(max_len, d_model)   # see positional-encoding.md
        self.norm_f = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab, bias=False)
        # Tie AFTER the head exists: same tensor, and it must be set before the
        # optimizer is constructed.
        self.head.weight = self.tok_emb.weight
        self.blocks = nn.ModuleList(
            TransformerBlock(d_model, n_heads, d_ff) for _ in range(n_layers)
        )
        self.d_model = d_model

    def forward(self, idx):
        B, T = idx.shape
        pos = torch.arange(T, device=idx.device)
        x = self.tok_emb(idx) * math.sqrt(self.d_model) + self.pos_emb(pos)
        for block in self.blocks:
            x = block(x)
        return self.head(self.norm_f(x))                 # (B, T, vocab) logits


if __name__ == "__main__":
    torch.manual_seed(0)
    m = TinyTransformer(vocab=1000, d_model=512, n_heads=8, d_ff=2048, n_layers=6)
    print(m(torch.randint(0, 1000, (2, 64))).shape)  # torch.Size([2, 64, 1000])
    print(sum(p.numel() for p in m.parameters()))    # 19_661_824

That last number is checkable by hand: tied tok_emb ($1000\times 512$) 512,000, plus pos_emb ($512\times 512$) 262,144, plus 6 blocks at $12d^2+4d=3,147,776$ each = 18,886,656, plus norm_f 1,024. Sum: 19,661,824. If your count is off by ~512,000 the head is not actually tied; if it is off by a factor of two, you changed d_ff.

The four knobs that matter, with the ranges you actually see:

Notes on the code that interviewers probe:

• out_proj is applied inside the residual add, not outside it. Adding before projecting changes the block’s function, not just its efficiency.
• self.head.weight = self.tok_emb.weight rebinds both to one tensor and must run before the optimizer is built. Two parameter groups with different LRs will then apply conflicting updates to the same weights.
• norm_f before the head is not optional in a pre-norm stack: the residual stream’s norm grows with depth, so without it the logit scale — and hence the effective softmax temperature — varies with depth.
• is_causal=True must match the training-time masking exactly. A model trained bidirectionally and served causally produces garbage from token one.

4. Numbers and trade-offs

Parameter count, derived

Per block, with $d_{ff}=4d$ and no biases:

Six encoder layers: $6\times 3.15\text{M}=18.9\text{M}$. Word embedding, tied to the output projection, at a 32k-token vocabulary: $32000\times 512=16.4\text{M}$. The encoder alone is therefore $\approx 35.3$ M. The decoder adds $6\times (4d_{\text{self}}^2+4d_{\text{cross}}^2+8d_{\text{mlp}}^2+6d)=6\times 4.20\text{M}=25.2\text{M}$, giving a full encoder–decoder base model of roughly $60$ M counting weights only. The familiar headline is ~65M, and the bulk of that gap is vocabulary, not biases: the paper’s BPE vocabulary is ~37k tokens rather than the 32k assumed here, and the extra $5000\times 512=2.56$M of embedding takes the derived total to ~63M. The biases this derivation omits and the fixed sinusoidal encoding (if you materialise it as a buffer) add only ~0.33M. Two observations worth carrying: the MLP is two-thirds of every block, and the embedding is nearly half of a small encoder. That share falls as $d$ grows, but not to zero: Llama-3-8B uses a 128,256-token vocabulary and untied input and output embeddings (tie_word_embeddings=False, as in Llama-2), so its single embedding matrix is $128256\times 4096=525$ M parameters — about 6.5% of an 8B model — and the input+output pair is $2Vd\approx 1.05$ B, ~13% of 8.03B. Those two are by far its largest single matrices.

Compute per token, derived

Count multiply-accumulates as 2 FLOPs and ignore the softmax (it is $O(n^2)$ with a tiny constant). Per layer, per token:

• Attention: QKV projections $3\cdot 2d^2$, output projection $2d^2$, plus the score and value matmuls at $2\cdot 2nd=4nd$. Total $8d^2+4nd$.
• MLP: $2\cdot 2\cdot d\cdot 4d=16d^2$.

So the per-token budget is $24d^2+4nd$, and the ratio of attention to MLP work is $\frac{4nd}{16d^2}=\frac{n}{4d}$. The crossover is $n=4d$. For the 2017 base model ($d=512$) that is 2048 tokens — at or above the length of most WMT sentence pairs — so 2017 was an MLP-dominated computation. For Llama-3-8B ($d=4096$) the crossover is 16384 tokens: at 8k context, attention is

$$\frac{4nd}{24d^2+4nd}{|}_{d=4096,n=8192}=\frac{134,217,728}{402,653,184+134,217,728}=0.25$$

of the FLOPs per token, so the MLP still dominates. This is the honest answer to “why is long context expensive?” — not because attention is inherently the compute cost, but because it is the only term that grows with $n$, and it grows quadratically.

Memory at inference: the KV cache, in context

The cache size is the formula derived in Attention §4: $2\times L\times n_{kv}\times d_{head}\times n\times \text{bytes}$, where $d_{head}=d_k$ is the per-head width and $n_{kv}$ is the number of KV heads (equal to the head count $h$ without GQA). Plug in the 2017 base model — 6 encoder plus 6 decoder layers, 8 heads, $d_k=64$, fp32 — and it is $2\times 12\times 8\times 64\times 4=49,152$ bytes/token $=48$ KiB/token (binary units: 1 KiB = 1024 B), about 192 MiB (201 MB) at 4096 tokens: trivial next to a modern model’s weights. The regimes where that stops being true, and the serving consequences, are in Attention §4 and vLLM and PagedAttention.

Which axis you are optimizing

5. Pitfalls

Post-norm with a pre-norm learning rate. Symptom: the loss spikes in the first few hundred steps and either recovers only with a warmup restart or diverges outright. The gradient at initialization near the output layer is simply too large for the LR you chose — Xiong et al. 2020 derives exactly this.

Forgetting the final norm in a pre-norm stack. Symptom: training is fine but the logits’ scale drifts with depth; you see the loss plateau at a value that looks like a calibration problem rather than a capacity problem, and sampling temperature has to be retuned per checkpoint.

Applying the residual add to the wrong tensor. x = x + mlp(norm2(x)) is correct; x = norm2(x + mlp(x)) is post-norm; x = x + mlp(x) (no norm) usually still trains at small scale and hides the bug. Symptom: no crash, a loss curve that is merely worse, and a model that is unusually sensitive to initialization scale.

Assuming the MLP is cheap. Symptom: profile shows the “attention” budget blown at 32k context and you optimize attention, while 80%+ of the FLOPs are in $W_1$/$W_2$ GEMMs. At $n<4d$ the MLP is the thing to optimize — quantization and MoE (see Modern Architecture Deltas) attack it directly.

Tying weights with two optimizer groups. Symptom: the loss increases for a few hundred steps and then recovers, or embedding norms diverge from head norms. The two parameter groups are applying conflicting updates to one tensor.

Assuming a causal model can be evaluated bidirectionally. Symptom: an elegant offline evaluation using masked-token prediction scores far above the model’s real generation quality, because the model has never seen a right context during training.

Treating $d_{ff}$ as a rounding detail. Symptom: you port a checkpoint, change $d_{ff}$ to a “nicer” number, and get NaN-free garbage — the shapes match and the weights do not.

6. Using it in a project

You almost never build a Transformer. What you actually control:

The checkpoint’s shape, not the architecture. Vocabulary size, $d$, $L$, $d_{ff}$, and head count are fixed when you download a model. Your only levers are which checkpoint, how many bits, and how it is served.

When NOT to use a Transformer block. Reach for something else when:

• $n$ is small and the structure is tabular. Gradient-boosted trees beat a Transformer on a 10k-row, 50-column tabular problem, they need no tokenization, and they train in seconds. The Transformer’s inductive bias buys nothing when there is no meaningful long-range order to model.
• Latency budget is single-digit milliseconds. A Transformer’s cost is dominated by $O(L)$ sequential block evaluations; a distilled CNN or a n-gram/feature-hashing model will fit in a budget the Transformer cannot.
• The task is a per-token classification with abundant labels and short inputs. A frozen encoder plus a linear head is usually the right call; a decoder-only LLM is a much more expensive way to get the same label.
• You need a hard guarantee of determinism and auditability. Transformer generations are stochastic functions of weights and sampling; if the domain demands a rule engine, no amount of prompting fixes it.
• The data is a few hundred labeled examples. Fine-tuning the block structure will overfit; few-shot prompting plus retrieval, or a linear probe on a frozen encoder, is the better first move.

When a Transformer is the right tool but the block is not the problem. If your RAG system gives bad answers, the block is nearly never at fault — retrieval recall, prompt construction, and output parsing are. See RAG. Optimizing the block before measuring those is the classic premature optimization.

Two block-level decisions you do get to make. Pre-norm vs post-norm when you train from scratch (choose pre-norm), and whether to use the fused attention kernel (choose it; F.scaled_dot_product_attention is a drop-in with the same math). Everything else is already decided by whoever trained the checkpoint.

7. How you would prove it worked

A Transformer has no product metric of its own. What you prove is that a component earns its place. Three experiments, in increasing cost:

The residual ablation. Train two identical small models, one with $\mathbf{x}+\mathrm{Sublayer}(\cdot )$ and one with $\mathrm{Sublayer}(\cdot )$ only. (example) Expect the no-residual run to plateau at a substantially higher loss and to be far more LR-sensitive. This is the one experiment in this chapter that is nearly always worth running, because it makes the central claim of the architecture measurable in an afternoon.

Pre-norm vs post-norm with warmup removed. Train both, both without warmup. (example) Post-norm shows an early loss spike and often diverges; pre-norm descends smoothly. Then add warmup back to post-norm and show it recovers — which is the demonstration that the norm placement was the variable, not the optimizer.

The mixing/thinking split. Ablate attention only, then the MLP only, at matched parameter count. (example) Removing attention hurts order-sensitive and retrieval-sensitive tasks much more than it hurts token-level tasks; removing the MLP hurts nearly everything. If your two curves look identical, you are evaluating on a task that does not depend on sequence order, and the ablation proves nothing. The cheap version of this experiment uses the permutation test from Positional Encoding §7 as the order-sensitive probe.

Reporting template (example — replace with your own numbers). “We compared pre-norm and post-norm 12-layer blocks on the same data at the same LR. Post-norm without warmup diverged at step ~400; with a 2000-step warmup it reached 2.61 validation loss. Pre-norm without warmup reached 2.58 and was stable, removing one hyperparameter and 2000 steps of schedule.”

8. Interview drill

“Explain a Transformer block in 60 seconds.” Strong answer: two sublayers — attention, which mixes information across positions, and a position-wise MLP, which transforms each position independently — each wrapped in a residual connection with a layer norm on the sublayer’s input. The residual stream is the state; the sublayers only add to it. Pre-norm placement is what makes deep stacks trainable without warmup. Common wrong answer: reciting the 2017 figure as “attention then layer norm then feed-forward then layer norm” without saying what the residual path is for. That is a diagram, not an explanation.

“Why is the residual connection load-bearing?” Strong answer: it makes the identity path between the input embedding and the output head unbroken, so gradients reach early layers unimpeded and information is added rather than overwritten — the “residual stream” reading. Removing it reintroduces depth-dependent gradient decay. Follow-up: “why does the norm then have to move inside the residual?” → because a norm on the residual path renormalises the identity path, which is exactly the post-norm instability Xiong et al. 2020 analyses.

“Why did the Transformer replace the RNN? Be specific.” Strong answer: self-attention has $O(1)$ sequential operations per layer and $O(1)$ maximum path length between positions, versus $O(n)$ and $O(n)$ for a recurrence. That removes the parallelism barrier on the time axis and the long-range gradient-decay problem at once, at the cost of $O(n^2)$ compute. Common wrong answer: “it has better accuracy.” Accuracy is an outcome; the architectural claim is about parallelism and path length, and both are exactly what made training on web-scale data feasible.

“What does the feed-forward network do that attention can’t?” Strong answer: it is the only nonlinear, per-position computation in the block. Attention only re-weights and averages existing vectors — it is a convex combination, so with softmax weights the output is bounded by the inputs’ convex hull. The MLP creates new features (an expanded hidden space, a nonlinearity, a projection back), and it holds roughly two-thirds of the block’s parameters. Common wrong answer: “it’s just a classifier” or “it adds depth.” It is a key-value memory: the up-projection’s rows act as learned feature detectors and the down-projection turns each on into a residual-stream direction, which is why the MLP — not attention — is the target of MoE and the layer that quantization damages most.

“Which is more expensive, attention or the MLP?” Strong answer: it depends on $n$ relative to $d$. They cross at $n=4d$ for $d_{ff}=4d$: below it the MLP dominates, above it attention does. For a 2017 base model at 512 tokens attention is ~14% of the FLOPs ($4nd/(24d^2+4nd)$); at 32k tokens it is ~91%. “Attention is the expensive part” is only true at long context. Common wrong answer: “attention, because it’s $O(n^2)$” — dropping the constant is fine for asymptotics and wrong for the regime you are actually in.

“Why layer norm and not batch norm?” Strong answer: layer norm computes statistics per position across features, so its behaviour is identical at training and inference and independent of batch size and padding. Batch norm couples positions in a batch, which is degenerate in autoregressive decode where the effective batch is 1, and it makes sequence padding statistically meaningful. Follow-up: “why does RMSNorm work?” → the re-centering term matters less than the re-scaling term empirically (Zhang & Sennrich 2019); RMSNorm keeps the rescaling and drops the mean and bias, saving parameters and a reduction.

“The 2017 paper scaled embeddings by $\sqrt{d}$ — why?” Strong answer: a scale calibration for the embedding’s initial statistics relative to the norm’s learned $\gamma$; with a norm immediately after, the forward output is invariant to it, so removing it costs a slightly worse early loss curve and nothing structural. Common wrong answer: “to prevent the positional encoding from dominating,” or “to keep gradients from vanishing.” Both are plausible-sounding and neither is what the paper’s own code does.

“What would you change in the block for a 2026 model?” Strong answer: pre-norm with RMSNorm, SwiGLU MLP with $d_{ff}\approx 8d/3$ (rounded for hardware), no biases, RoPE instead of additive positional encoding, GQA to shrink the KV cache, and a fused FlashAttention kernel. Each of those is a response to a specific cost — see Modern Architecture Deltas.

Next: Attention — the mixing half of the block, derived rather than asserted: why the query–key dot product, why $\sqrt{d_k}$, and why multiple heads.