BERT vs GPT vs Encoder-Decoder

TL;DR — Three attention patterns, three objectives. BERT is bidirectional with masked language modeling: predict the tokens you hid, seeing both sides. GPT is causal with next-token prediction: predict each token from the prefix only, which makes a single loss factorize the whole joint distribution and gives you generation for free. T5/BART stack both and add cross-attention, so the encoder reads the source once and the decoder attends to it while generating. Decoder-only won the general-purpose crown because one objective covers every task expressed as text and scales cleanly; encoder-only still owns embedding, reranking and classification, because bidirectional context produces a better fixed representation per token — and nothing you can generate with.

1. Why this exists

Pretraining is expensive, and the objective decides what the pretrained weights are worth. Three concrete failures motivate the split.

A model trained to generate cannot be asked to read a sentence whole. If you train with a causal mask, position $t$ sees only $\mathrm{1..}t-1$. Take the sentence “The trophy would not fit in the brown suitcase because it was too ___” and ask for the referent of “it” — an encoder that sees the whole sentence can use the following word “suitcase”; a causal model at the word “it” cannot. This is not a quality argument but a masking argument: the architecture forbids the information, so no amount of scale recovers it at that position.

A model trained to fill in blanks cannot be sampled from coherently. Masked language modeling corrupts the input by replacing tokens with a sentinel and predicts the originals independently — each masked position is predicted from the same corrupted context, with no dependency between the predictions. You cannot write down $p(x_1,\dots ,x_n)$ from that, so you cannot autoregressively generate. Bidirectional encoders are very good at scoring and very bad at writing.

A model trained on next-token prediction has no place to put the source. When the input and output are genuinely different sequences drawn from different distributions — source German to target English; a document to its summary — a decoder-only model handles it by concatenating them into one stream and trusting the causal mask to keep them apart. That works, but it spends the decoder’s capacity re-encoding the source. An encoder that reads the source once and cross-attends to it is the structurally cheaper arrangement, and it is why T5 and BART held the seq2seq crown long after GPT-style models took over general-purpose text.

What you are choosing. Not “which model is smarter” but which conditional distribution the loss is a maximum-likelihood fit of, which in turn fixes the set of operations the model can perform well. Generation requires a factorized joint. Scoring requires bidirectional context. Seq2seq wants both, separated.

2. How it works

2.1 Three attention patterns

The architectures differ in exactly one mechanical thing: which positions the attention mask allows, and where the queries come from.

flowchart TB
    subgraph ENC["Encoder-only (BERT)"]
        E1["tokens"] --> E2["bidirectional self-attention × L"]
        E2 --> E3["contextual embeddings"]
    end
    subgraph DEC["Decoder-only (GPT)"]
        D1["prefix tokens"] --> D2["causal self-attention × L"]
        D2 --> D3["next-token distribution"]
    end
    subgraph ED["Encoder-decoder (T5/BART)"]
        C1["source tokens"] --> C2["bidirectional encoder × L"]
        C2 --> C3["encoder states (K, V)"]
        C4["output so far"] --> C5["causal decoder × L"]
        C3 --> C5
        C5 --> C6["next-token distribution"]
    end

Three mask shapes, drawn as $n\times n$ attention patterns (■ = allowed):

$$\text{bidirectional}=\left(\begin{array}{cccc}■& ■& ■& ■\\ ■& ■& ■& ■\\ ■& ■& ■& ■\\ ■& ■& ■& ■\end{array}\right)\text{causal}=\left(\begin{array}{cccc}■& & & \\ ■& ■& & \\ ■& ■& ■& \\ ■& ■& ■& ■\end{array}\right)$$

The third shape is cross-attention: the queries come from the decoder and the keys and values come from the encoder, so the rectangle is $n_{dec}\times n_{enc}$ and it is not triangular — every decoder position may look at the entire source. The mask mechanics are unchanged from Attention; only which axis is masked differs.

2.2 The masked language modeling loss

BERT corrupts a fraction of positions and predicts the originals. Let $M$ be the set of masked positions and $\tilde{x}$ the corrupted sequence:

$${\mathcal{L}}_{\mathrm{MLM}}=-\sum _{i\in M}\log p_{\theta }\left(x_i\mid \tilde{x}\right)$$

• $x=(x_1,\dots ,x_n)$ — the original token sequence, already subword-tokenized.
• $\tilde{x}$ — the corrupted input. BERT’s recipe: choose 15% of positions uniformly; of those, replace 80% with [MASK], 10% with a uniformly random vocabulary token, and leave 10% unchanged. The random and unchanged branches exist because [MASK] never appears in downstream text — a model trained to expect it would face a train/test mismatch. The cost is that some targets are deliberately unlearnable noise, which is accepted.
• $M$ — the set of masked positions, $|M|\approx 0.15n$. The sum runs only over masked positions. This is the load-bearing detail for every efficiency claim about MLM: 85% of the positions contribute no gradient at all. ELECTRA names this directly as the reason MLM is compute-inefficient — its replaced-token-detection task is defined over all input tokens rather than “just the small subset that was masked out”.
• $p_{\theta }(x_i\mid \tilde{x})$ — the model’s distribution over the vocabulary at position $i$, conditioned on the entire corrupted sequence, left and right. This is where bidirectionality enters: nothing in the mask forbids position $i-1$ or $i+1$, so both contribute to the representation used to predict $x_i$.

What this loss actually is. It is a sparse reconstruction objective with an independence assumption baked in: the model maximizes ${\sum }_{i\in M}\log p(x_i\mid \tilde{x})$, which is not a factorization of $\log p(x)$ — no chain rule appears anywhere. Removing the masking uses the true context on both sides, but you cannot sample from what you trained, because you never modelled a joint. Getting this distinction right is the usual first filter in interviews.

Why 15%. Prediction and representation are in tension: mask more and the corrupted context yields worse representations, mask less and too few positions produce gradient. 15% is BERT’s empirical setting from an ablation, not a derivation.

2.3 The causal language modeling loss

GPT factorizes the joint distribution with the chain rule and fits each factor:

$${\mathcal{L}}_{\mathrm{CLM}}=-\sum _{t=1}^n\log p_{\theta }\left(x_t\mid x_1,\dots ,x_{t-1}\right)$$

• $x_t$ — the target token at position $t$.
• $x_{<t}=x_1,\dots ,x_{t-1}$ — the prefix. The causal mask makes this the entire conditioning set; position $t$’s query cannot attend to $t$ or later.
• $p_{\theta }(x_t\mid x_{<t})$ — the output distribution after the final projection to vocabulary size, softmaxed.

Because ${\prod }_{t=1}^{n}p(x_t\mid x_{<t})=p(x_1,\dots ,x_n)$ exactly, this one loss is a proper sequence likelihood. That buys three things MLM cannot have: generation (sample $x_t$, append it, repeat), arbitrary conditioning (prefix anything and the model continues it), and a signal on every position.

The price is that the model never sees the future while representing a position, so an encoder’s representation of token $i$ is strictly more informed. Encoder models pay the opposite price: better representations, no joint.

2.4 Encoder-decoder: span corruption and cross-attention

T5 unifies everything as text-to-text and replaces MLM with span corruption: contiguous spans of the source are replaced by unique sentinel tokens, and the decoder must emit the removed spans. Concretely, with spans (1,2) and (3) removed:

encoder input : Thank you <X> me to your party <Y> week .
decoder target: <X> for inviting <Y> last <Z>

• <X>, <Y>, <Z> — sentinel tokens, one per corrupted span. They are content-addressable markers: the decoder knows where in the original sequence to place what it generates, which is what lets one pass emit several non-adjacent spans.
• The decoder loss is ordinary causal language modeling over the target, with the encoder states injected through cross-attention at every layer.
• Why spans rather than independent positions? Predicting a contiguous span requires modelling within-span dependencies, so the objective is closer to a real sequence likelihood than MLM’s independent-blank-filling, and the target sequence is shorter than the source (the encoder sees the whole input, the decoder emits only the removed spans), which shifts compute to the encoder — the right place, since the encoder is the only fully parallel half.

BART takes a different route to the same architecture — a denoising autoencoder with several noise functions (token deletion, infilling, sentence permutation, document rotation) and a standard next-token decoder loss — but the shape is the same: bidirectional encoder, causal decoder, cross-attention between them.

2.5 Why decoder-only won for general-purpose models

Five reasons, in rough order of weight.

Universal task interface. Every task becomes “continue the text.” Classification is a label in the continuation; extraction is JSON in the continuation; translation is the translation. No task-specific head, no new architecture per task — BERT needs a fresh classification head fine-tuned per task, which means a separate training run and a separate artifact per task.

In-context learning emerged with scale, and only on generative models. GPT-3 demonstrated strong few-shot performance without any gradient updates, with tasks specified purely by text in the prompt. The mechanism is not available to an encoder: an MLM model cannot produce the demonstration completions that the prompt’s structure relies on.

Generation is native and is the floor of every agentic capability. Tool calling, chain-of-thought, structured output, multi-turn dialogue — all require producing tokens. An encoder-only model scores; it does not act.

One stack, one loss, every parameter used at every token. There is no reconstruction-vs-target asymmetry to tune, no two-stack balance, no cross-attention to schedule. This is a boring reason and probably the decisive one at scale: the scaling-law literature and the engineering tooling were built on this configuration, and everything downstream was optimized for it.

Post-training fits it naturally. Instruction tuning and preference optimization (SFT, RLHF, DPO, GRPO) are all defined as improving a generative distribution on prompt–completion pairs. That framing has no analogue on a scoring model.

Historical honesty. Encoder-decoder held its own through roughly 2021 and is still the better structure for true sequence-to-sequence work. Decoder-only won the general-purpose crown, not every crown.

2.6 What still wants bidirectional encoders

The rule of thumb: generate ⇒ decoder(-only); embed, score, or classify ⇒ encoder-style. The reason is structural — the pooled or per-token output of a bidirectional encoder is a better fixed representation, because every token was built from full context.

The trade is latency and cost, not quality: an encoder is a single forward pass, while an instruction-tuned decoder used as an embedder or reranker must generate and is orders of magnitude slower at the same quality. The line does blur — GTE-Qwen and E5-Mistral are decoder-based embedders, and LLM rerankers exist — but they buy their flexibility with inference cost, which is the wrong trade for high-QPS retrieval.

Why embeddings specifically. A bi-encoder produces one vector per document independently of the query. Pooled over a causal model’s states, the mean mixes early positions that saw only a prefix with late ones that saw everything. Bidirectional attention removes the asymmetry, so any pooling scheme draws from equally-informed states. That is the whole argument for encoder embedders.

3. Implementation

Two fine-tuning shapes and one objective, against PyTorch 2.14 and transformers 5.19 [as of 2026-10].

Sequence classification on a frozen encoder

import torch
import torch.nn as nn
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")
enc = AutoModel.from_pretrained("bert-base-uncased")   # bidirectional, no LM head


class SequenceClassifier(nn.Module):
    """One linear layer on the [CLS] state: the whole BERT fine-tuning recipe."""
    def __init__(self, d_model, n_labels, dropout=0.1):
        super().__init__()
        self.drop, self.out = nn.Dropout(dropout), nn.Linear(d_model, n_labels)

    def forward(self, input_ids, attention_mask):
        h = enc(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state
        # (B, T, d_model). Position 0 is [CLS]; mean-pooling over real tokens is
        # the common alternative and is usually within noise of [CLS].
        return self.out(self.drop(h[:, 0]))


model = SequenceClassifier(d_model=enc.config.hidden_size, n_labels=2)
batch = tok(["a great film", "a dull film"], return_tensors="pt", padding=True)
logits = model(batch["input_ids"], batch["attention_mask"])

Three details that matter: attention_mask must be passed, or the model attends to [PAD] tokens; position 0 is [CLS] by convention, and using mean pooling instead is a real choice you should state; and the encoder is trained without a causal mask, which you cannot change after the fact.

Causal LM loss, written out rather than imported

import torch.nn.functional as F
from transformers import AutoModelForCausalLM

lm = AutoModelForCausalLM.from_pretrained("gpt2")
ids = torch.tensor([[1, 2, 3, 4, 5]])          # (B, T)


def causal_lm_loss(model, ids):
    """Shift by one: logits at t-1 score the token at t. Mean over T-1 positions."""
    logits = model(ids).logits                                  # (B, T, vocab)
    return F.cross_entropy(logits[:, :-1, :].reshape(-1, logits.size(-1)),
                           ids[:, 1:].reshape(-1))


print(float(causal_lm_loss(lm, ids)))

The shift is the entire implementation of “predict the next token”. Getting it wrong by one position produces a model that is fluent and subtly incoherent, which is why libraries expose it as a warning rather than silently fixing it.

Masked LM loss, written out

def mlm_loss(model, ids, tok, mask_ratio=0.15):
    """BERT corruption. Loss is the mean over corrupted positions ONLY."""
    masked = torch.rand(ids.shape) < mask_ratio
    roll = torch.rand(ids.shape)                # shared across positions
    corrupted = ids.clone()
    corrupted[masked & (roll < 0.8)] = tok.mask_token_id
    replaced = masked & (roll >= 0.8) & (roll < 0.9)   # 10% random token
    corrupted[replaced] = torch.randint_like(
        ids, low=0, high=model.config.vocab_size)[replaced]
    # The remaining 10% of chosen positions are left unchanged, by design.
    logits = model(corrupted).logits                            # (B, T, vocab)
    return F.cross_entropy(logits[masked], ids[masked])         # denominator = |M|

Note the difference: logits[masked] selects only corrupted positions, and F.cross_entropy’s default reduction="mean" divides by $|M|\approx 0.15n$, not by $n$. So roughly 15% of the sequence contributes gradient — exactly the inefficiency ELECTRA attacks, and the reason any MLM cost estimate that assumes every token is a training example is wrong by about 6.7×.

The knob that matters

4. Numbers and trade-offs

What each objective buys per unit of compute

The ELECTRA row is the interesting one: it gets bidirectional context and a loss on every token by changing the task to binary detection (was this token replaced by a generator?). The reported result is that the approach “substantially outperforms” MLM at the same model size, data and compute, with the largest gains for small models — including a single-GPU, 4-day model that beats GPT on GLUE at 30× the compute (ELECTRA). If you are pretraining an encoder on a budget, this is the row that matters; if you are fine-tuning an existing checkpoint, it is a footnote.

Cost per token, and why encoder inference is cheap

An encoder performs one forward pass over the sequence, in parallel, with no KV cache needed and no autoregressive loop. A decoder generating $k$ tokens performs $k$ sequential forward passes, each re-reading the weights and the KV cache from HBM. So for the same model size and sequence length, embedding 10,000 documents with an encoder is one batched parallel pass each; “embedding” with a generative decoder is a generation loop. This — not any quality gap — is why encoder embedders remain the default for high-throughput retrieval.

Where the parameters go

Roughly: attention is $4d^2$ per layer, the FFN is $8d^2$ per layer at SwiGLU’s $8d/3$ hidden size (see Modern Architecture Deltas), so the FFN is about two-thirds of the block. In an encoder-decoder, both stacks carry that cost, plus cross-attention (another $4d^2$ per decoder layer), so a T5-style model at a fixed parameter budget has less depth per stack than a decoder-only model of the same size. That is a real capacity argument for decoder-only at equal parameters, and it is separate from the task-interface argument.

Sequence length

Encoders are overwhelmingly trained and used at 512 tokens. Decoders trained for long context are used at 4k–1M+. This is not a coincidence: the encoder’s unit of work is a passage to be scored, while the decoder’s is a conversation or document to be continued. If you push an encoder embedder to long sequences, you are using it outside its pretraining distribution and should expect degradation — which is why long-context embedding models are a separate, explicitly-trained family (and why late chunking exists as a technique at all).

5. Pitfalls

Masking with the wrong sentinel at inference. Symptom: fine-tuned encoder quality is far below the reported number, and the input contained no [MASK] tokens at all — because [MASK] was never in the downstream distribution and your model learned to key on it during pretraining. This is also why BERT’s 10% unchanged branch exists. For classification there is no masking at inference, so this only bites if you mimic the pretraining format.

Forgetting the causal shift by one. Symptom: the model generates text that is locally fluent and globally wrong — it has learned to copy the current token. If the loss drops suspiciously fast and generation repeats the prompt’s last token, check the shift before anything else.

Averaging causal states for embeddings. Symptom: retrieval recall is noticeably worse than a same-size encoder model on the same data. Mean-pooling a causal model’s last_hidden_state mixes early positions that saw almost nothing with late positions that saw everything. Use the last position’s state, or use a properly trained embedding model.

Treating [CLS] and mean pooling as equivalent. Symptom: a retrieval or clustering task underperforms despite good classification numbers. [CLS] was trained to aggregate the sentence (BERT jointly with next-sentence prediction, a task later found weak and dropped by RoBERTa/ALBERT), so it is a sentence summary, not a metric-space embedding. Normalize embeddings and use the pooling the model card specifies.

Assuming MLM trains on the whole sequence. Symptom: your compute estimate is ~6× optimistic, or you compare an MLM loss to a CLM loss and conclude MLM learns faster. HuggingFace’s DataCollatorForLanguageModeling sets the unmasked labels to -100, and CrossEntropyLoss with ignore_index=-100 and reduction="mean" averages over the non-ignored targets only — so the reported loss is per-masked-token and needs no denominator “fix”. The genuine pitfall is that an MLM loss is not comparable to a CLM loss over all tokens: different denominator ($|M|$ masked positions versus all $n$ positions) and a different objective (denoising a corrupted sequence versus next-token prediction). Compare at equal tokens seen and equal wall-clock instead.

Comparing MLM and CLM at equal steps. Symptom: an apparently meaningful conclusion that “MLM learns faster”. MLM sees fewer loss terms per sequence and the two losses are on different scales (per-position cross-entropy over 15% of tokens versus over all tokens). Compare at equal tokens seen and equal wall-clock, and report both.

Fine-tuning an encoder at the head’s learning rate. Symptom: pretrained representations are destroyed in the first few hundred steps, and the model performs worse than a linear probe. The body needs 1e-5–3e-5; the head tolerates 1e-3. Use a smaller LR for the encoder or a separate parameter group.

Using a decoder-only model as a reranker at scale. Symptom: p99 latency dominated by reranking. A generative reranker costs a generation per candidate; a cross-encoder costs one parallel forward pass. At 100 candidates the difference is an order of magnitude, and it shows up as a p99 problem, not an average one.

6. Using it in a project

Reach for encoder-only when: you are embedding, reranking, classifying, extracting spans, or scoring. The questions to answer are the usual retrieval ones — is the vector normalized, is the query prefix required, what is the pooling — and the compute is one parallel pass. Concretely: a document vector used for nearest-neighbour search must be produced by a model whose pretraining objective is consistent with that use. See Embeddings and Retrieval, Hybrid Search and Reranking.

Reach for decoder-only when: the task is generative, open-ended, multi-turn, or agentic, or when you want one model to cover many tasks without per-task heads. This is now the default for anything an end user talks to.

Reach for encoder-decoder when: the task is genuinely sequence-to-sequence and the source is long relative to the target — translation, summarization, speech recognition, structured extraction from a document. The structural advantage is real: the encoder reads the source once, in parallel, and every decoder step attends to the same states rather than re-deriving them through the causal mask. Whisper is the clearest modern example: audio encoding is a bidirectional problem and the output is a sequence.

When NOT to use an encoder-only model:

• Any generation task. It cannot do it. Not “does it poorly” — the objective is not a joint distribution, so there is nothing to sample from.
• Open-ended instruction following. A classification head over a prompt is not instruction following, and adding one per instruction is not a strategy.
• Anything requiring in-context learning. The mechanism needs generated demonstration completions.
• Retrieval where the corpus and queries are adversarial or domain-shifted and a general encoder was not trained on it. Fine-tune the encoder or use a cross-encoder; a bigger general encoder is often the wrong purchase.

When NOT to use a decoder-only model:

• High-QPS embedding or reranking. The generation loop is the cost.
• Bit-exact scoring of a fixed pair. A cross-encoder gives you a single comparable score; a generative model gives you a distribution over labels whose calibration you now have to manage.
• Very small devices. A 100M-parameter encoder is a viable on-device classifier; a useful generative decoder is not, for the same quality.

When NOT to use an encoder-decoder: when the target is not appreciably shorter than the source, or when the task is better expressed as continuation. Summarizing a 10-token sentence with a T5-large is using a two-stack model where a conditional continuation would do.

7. How you would prove it worked

Ablation 1 — is your embedding model’s bidirectional context doing anything? Take a bidirectional encoder and a same-size causal model, both pretrained, and evaluate retrieval recall@10 on your corpus with identical pooling (last-token for the causal model, mean for the encoder). Baseline: the encoder. Prediction: the encoder wins, and the gap narrows for short passages where the causal model’s last token has effectively seen the whole input. Falsified if the causal model matches the encoder at every passage length — then on your data the bidirectionality is not the binding constraint, and you should prefer the one model that also generates.

Ablation 2 — mask ratio. Pretrain two small encoders at mask_ratio 0.15 and 0.30, identical otherwise; fine-tune both on the same downstream task. Metrics: final task accuracy and tokens/sec during pretraining. Baseline: 0.15. Prediction: 0.30 reaches a lower pretraining loss per step but a worse downstream score, because the corrupted context is harder to build representations from. Falsified if 0.30 wins downstream — report it, because the 15% figure is an empirical result from 2018, not a law.

Ablation 3 — the shift bug, deliberately. Train two causal models for a few thousand steps, one with the correct one-position shift and one predicting the current token. Metric: held-out loss and a generation sample. Baseline: the correct shift. Prediction: the broken one reports a much lower loss and generates degenerate copies. This is worth running once, because it calibrates how much a loss number can lie about a generation system — and that calibration is what interviewers are probing when they ask you to justify an eval.

Ablation 4 — decoder-only vs encoder-decoder on a summarization task. Same parameter budget, same data, one decoder-only model with the document in the prompt and one encoder-decoder. Metrics: ROUGE or an LLM-judge score, plus TTFT and tokens/sec at a fixed batch. Baseline: the decoder-only. Prediction: quality within noise at short documents; the encoder-decoder pulls ahead as source length grows, and its prefill cost is lower because the source is processed once in parallel. Falsified if the decoder-only matches quality and cost at long source lengths — in which case the two-stack structure is not buying what it costs on your workload.

Reporting template (example — replace with your own numbers). “We replaced a generative reranker with a cross-encoder over the same top-100 candidates: nDCG@10 was unchanged (0.71 → 0.70, example) while p99 latency fell from 3.1 s to 180 ms (example). We kept the generative path for the top-5 only, where we want an explanation alongside the score.”

8. Interview drill

“Why can’t you generate with BERT?” Strong: the MLM loss sums $\log p(x_i\mid \tilde{x})$ over masked positions independently — it never models a joint over the sequence, so there is no $p(x_1,\dots ,x_n)$ to sample from. Generation needs a factorization, and the chain rule is the one GPT uses. Wrong: “it’s bidirectional, so it can’t do next-token prediction.” You could add a causal head to an encoder stack, but it would have been trained with a bidirectional mask and would be leaking the answer; the blocker is the objective, not the architecture alone.

“Why 15% masking?” Strong: a trade-off between how much gradient you get per sequence and how badly you corrupt the context used to build representations. It is BERT’s empirical setting from an ablation, not a derivation, and it is why the loss touches only ~15% of positions. Wrong: “because 15% of tokens is enough information” — not a mechanism. Also wrong: treating 15% as the mask rate while ignoring that only 80% of those become [MASK].

“Why did decoder-only win?” Strong: one objective factorizes the joint, so it generates; every task is expressible as continuation; in-context learning emerged with scale only on generative models; and the loss uses every token, which makes it the simplest thing to scale and post-train. Wrong: “it’s just better.” Encoder-decoder is still better for true seq2seq, and encoders are still better for embedding. The win is scope-specific.

“Which objective trains on the most positions per sequence?” Strong: causal LM, which puts a loss on all $n$ (or $n-1$) positions; MLM uses ~15%. ELECTRA’s replaced-token detection gets a signal on every token with bidirectional context by changing the task, which is why it is more sample-efficient than MLM at equal compute. Wrong: “MLM, because it looks at both directions” — direction of context is not the number of loss terms.

“Why are encoder models still used for embeddings?” Strong: a bidirectional encoder conditions every position on the full document, so any pooling draws from equally-informed states. Pooling over a causal model’s states mixes early positions that saw only a prefix with late ones that saw everything. Plus one parallel forward pass versus an autoregressive loop. Wrong: “encoders are smaller/faster” — that is a consequence of the deployment choice, not the reason the representations are better.

“What is cross-attention, and which axis is masked?” Strong: queries from the decoder, keys and values from the encoder, so the score matrix is $n_{dec}\times n_{enc}$ and not triangular — the whole source is visible to every decoder position. The causal mask still applies within the decoder’s own self-attention. Wrong: “the encoder attends to the decoder” — inverted, and a common slip.

“Why does T5 use span corruption instead of MLM?” Strong: replacing contiguous spans makes the decoder model within-span dependencies, which is closer to a real sequence likelihood than filling independent blanks; and the target is shorter than the source, moving compute to the fully parallel encoder. Wrong: “spans are harder, so the model learns more” — not a mechanism, and it does not explain the length asymmetry.

“Can you fine-tune BERT as a generator?” Strong: you can attach a causal decoder (that is roughly BART’s shape), but then you are training a new model, not fine-tuning BERT. Fine-tuning reuses the pretrained objective’s inductive bias; a different objective needs the pretraining compute again. Wrong: “yes, just add a language modeling head” — the head is trivial; the pretrained representations were never trained to support it.

Next: Pretraining Pipeline — the objectives above assume you have tokens. The next chapter is where those tokens come from: crawl, filter, deduplicate, tokenize, and the distributed loop that consumes them.