Positional Encoding

TL;DR — Attention is permutation-equivariant: shuffle the tokens and the output rows shuffle with them, so “dog bites man” and “man bites dog” are the same input. Position must therefore be injected explicitly. Three families exist: add a vector to the embedding (sinusoidal, fixed; learned, a table), add a bias to the attention logits (ALiBi), or rotate the query and key vectors by an angle proportional to position (RoPE). RoPE won because $R_m^{\top }{R}_n=R_{n-m}$ — absolute positions cancel and attention scores become a function of the relative offset, with zero extra parameters, zero extra activation memory, and no change to attention’s form. Extending context is then a matter of rescaling the frequencies or the position indices.

1. Why this exists

Write out what self-attention does to a sequence, ignoring the projections:

$$y_i=\sum _j\frac{\exp (q_i\cdot k_j)}{{\sum }_{j^{\prime }}\exp (q_i\cdot k_{j^{\prime }})}{v}_j$$

Nothing in that expression refers to $i$ or $j$ except as indices into the sum. Apply a permutation $\pi$ to the input rows and every output row is permuted the same way: $\mathrm{Attn}(PX)=P\mathrm{Attn}(X)$. The model is permutation-equivariant — it is, to a first approximation, a very expensive bag of words.

That is a hard failure, not a stylistic one. “The server rejected the request because the token expired” and “the token rejected the server because the request expired” produce identical attention outputs. Any task whose answer depends on order — basically every task worth doing — is unreachable without position information. The failure is also silent: the model trains, the loss goes down, and it learns to answer order-insensitive questions well enough to look like it is working. Symptom: your model does fine on topic classification and fails on any question whose answer is a sequence of instructions, and fine-tuning on more data does not fix it.

The demo that makes this concrete takes two minutes and is worth running before you read the rest of this chapter:

# transformers==4.44.0, torch==2.4.0
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()

def logprob_of(text: str, continuation: str) -> float:
    ids = tok(text, return_tensors="pt").input_ids
    cont = tok(continuation, add_special_tokens=False, return_tensors="pt").input_ids
    full = torch.cat([ids, cont], dim=1)
    with torch.no_grad():
        logits = model(full).logits
    lp = torch.log_softmax(logits[0, len(ids[0]) - 1:-1], dim=-1)
    return lp.gather(-1, cont[0].unsqueeze(-1)).sum().item()

print(logprob_of("The dog bites the man.", " The reverse is"))
# (example) clearly higher after a correct forward pass than:
print(logprob_of("The man bites the dog.", " The reverse is"))

Now imagine the same model with no positional encoding at all: both prompts would produce identical logits, and the difference above would be exactly zero. That is what §1 formalizes.

2. How it works

2.1 The three places you can put position

There are only three places a positional signal can enter a Transformer, and every published scheme is one of them:

1. Added to the token embedding, before the first block: $x_i\leftarrow x_i+p_i$. (Sinusoidal, learned absolute, and every “position embedding” variant.)
2. Added to the attention logits, per head, at score time: $s_{ij}\leftarrow s_{ij}+b(i,j)$. (ALiBi, T5 relative bias, Shaw et al. 2018.)
3. Mixed into $q$ and $k$ themselves, as a function of position. (RoPE — a rotation, not an addition.)

The distinction between (1) and (3) is the one that took the field five years to absorb. In (1), position enters through the content: the vector the model sees at position $i$ is a different vector, and everything downstream — $W_Q$, $W_K$, $W_V$, the MLP — has to learn to disentangle “this is about the token” from “this is about the position.” In (3), position enters only through the $q\cdot k$ product, which is exactly where the model needs it and nowhere else.

2.2 Sinusoidal encodings: what each symbol is for

$${\mathrm{PE}}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d}}\right),{\mathrm{PE}}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d}}\right)$$

• $pos\in \{0,1,\dots ,n-1\}$ — the position index. It is integer and unbounded: the function is defined for any $pos$, which is precisely why it was chosen over a lookup table.
• $i\in \{0,\dots ,d/2-1\}$ — the frequency index. Each value of $i$ selects a different sinusoid, and the pair $(2i,2i+1)$ shares one frequency as its sine and cosine. Note that $i$ indexes frequency pairs, so there are $d/2$ frequencies across $d$ dimensions.
• $2i/d$ — the frequency exponent. It maps the frequency index into $[0,1)$, which makes the divisor $10000^{2i/d}$ run geometrically from $10000^0=1$ (fastest) to $10000^{1-2/d}\approx 10000$ (slowest). The paper states the consequence directly: “the wavelengths form a geometric progression from $2\pi$ to $10000\cdot 2\pi$.” So the fastest dimension rotates once every ~6.3 tokens and the slowest once every ~62,832 tokens. This is the multi-scale clock idea: fine dimensions resolve local order, coarse dimensions resolve position in the document, and using a geometric spacing rather than a linear one is what means no single scale has to carry the whole range.
• $10000$ — the base, a magic constant in 2017 that became a tuned hyperparameter with RoPE (LLaMA-3 raises it to 500,000; see §2.5). Raising it stretches every wavelength by the same factor, which is how you buy context length.
• The addition $\mathrm{PE}(pos)$ to the embedding is itself load-bearing and is the part people get wrong. Because the pair is $(\sin \theta ,\cos \theta )$ for the same $\theta$, the encoding vector at position $pos+k$ is a fixed linear map (a block-diagonal rotation, one $2\times 2$ block per frequency pair) applied to the encoding at $pos$. In other words $\mathrm{PE}(pos+k)=R_k\mathrm{PE}(pos)$ for a $k$-dependent rotation, independent of $pos$. That property is the entire justification the paper gives — “we hypothesized it would allow the model to easily learn to attend by relative positions.” A learned table has no such structure: there is no guarantee that $p_{m+k}$ relates to $p_m$ at all, so the model must discover relative position from scratch from data.

2.3 Why RoPE: the derivation in one line

RoPE starts from a requirement rather than a construction. The requirement is

$$⟨f_q(x_m,m),f_k(x_n,n)⟩=g(x_m,x_n,m-n)$$

That is: the attention score between a query at absolute position $m$ and a key at absolute position $n$ must depend on their positions only through the difference $m-n$. Su et al. 2021 (RoFormer, https://arxiv.org/abs/2104.09864, Eq. 11) derive that in 2D the only solution of this form is a rotation.

Concretely: pair up the dimensions of each head vector, $(x_{2i},x_{2i+1})$, and treat each pair as a complex number. Multiply by $e^{im{\theta }_i}$ — i.e. rotate by angle $m{\theta }_i$:

$$R_{m,i}=\left(\begin{array}{cc}\cos (m{\theta }_i)& -\sin (m{\theta }_i)\\ \sin (m{\theta }_i)& \cos (m{\theta }_i)\end{array}\right),{\theta }_i={\Theta }^{-2i/d_k}$$

• $m$ — the absolute position of the query (or $n$ for the key). It appears only inside the rotation angle.
• ${\theta }_i$ — the rotation frequency of dimension pair $i$. It is the exact analogue of $10000^{-2i/d}$ in the sinusoidal formula: the same geometric progression, with fast pairs resolving local order and slow pairs resolving global order.
• $\Theta$ — the base, 10,000 in the paper (and in LLaMA-1/2). Because ${\theta }_i={\Theta }^{-2i/d_k}$, raising $\Theta$ decreases every ${\theta }_i$ and therefore stretches every wavelength — the knob long-context extension turns.
• $d_k$ — the per-head dimension. Note it is $d_k$, not $d_{model}$: RoPE is applied per head, to the projected $q$ and $k$, never to the residual stream. Getting this wrong (rotating the $d_{model}$-wide hidden state) changes which dimensions pair with which.

The identity that makes it work:

$$(R_{m}q{)}^{\top }(R_{n}k)=q^{\top }{R}_m^{\top }{R}_{n}k=q^{\top }{R}_{n-m}k$$

because rotations compose additively: $R_m^{\top }{R}_n=R_{n-m}$. The absolute positions $m$ and $n$ have cancelled; what survives is the relative offset. This is why RoPE is a relative position scheme implemented with absolute angles — the property that took the field a while to appreciate.

Three consequences follow:

• No parameters. ${\theta }_i$ are constants. Compare a learned relative position table $b(i,j)$ (T5) or per-head relative vectors (Shaw et al. 2018), which add parameters and, worse, are undefined for offsets outside the trained range.
• No added activation memory. The rotation is applied in place to $q$ and $k$ at each layer. An additive encoding materialises an $n\times d$ matrix first.
• The fast/slow split gives natural decay. The paper (§3.3, “long-term decay”) shows that with geometrically spaced ${\theta }_i$ the expected inner product between rotated $q$ and $k$ decays as $|m-n|$ grows — a free prior that nearby tokens are more likely to be related.

2.4 One detail that trips up every implementation: which pairs

Mathematically the pairs are $(x_0,x_1),(x_2,x_3),\dots$ with ${\theta }_i={\Theta }^{-2i/d_k}$. The efficient implementation (Su et al. §3.4.2) uses the half-split form instead:

$$q\leftarrow q\odot \cos (m\theta )+rotate\_half(q)\odot \sin (m\theta )$$

where $rotate\_half([q_0,\dots ,q_{d/2-1},q_{d/2},\dots ,q_{d-1}])=[-q_{d/2},\dots ,-q_{d-1},q_0,\dots ,q_{d/2-1}]$ and $\theta =({\theta }_0,\dots ,{\theta }_{d/2-1})$ is repeated twice to fill $d$ dimensions. The two forms are related by a fixed permutation of dimensions, so they compute the same function up to a basis change — but a checkpoint trained with one convention and loaded with a kernel that assumes the other produces silently wrong logits, not an error. This is the single most common RoPE bug, and §5 lists its symptom.

2.5 Long context is a RoPE-frequency problem

Every long-context extension is an intervention on ${\theta }_i$ or on the position index $m$:

The pattern: PI scales all frequencies equally and damages local resolution; NTK-aware scaling exploits the fact that you only need the slow dimensions to cover the new range, leaving the fast ones alone. The honest summary [as of 2026-02] is that frontier models train natively at longer context and use YaRN/NTK only for further stretching; the RoPE base is raised, not the positions rescaled.

flowchart TB
    SUB["q, k after projection (per head)"] --> PAIR["Treat as d_k/2 complex pairs"]
    PAIR --> ROT["Rotate pair i by angle m·θ_i<br/>θ_i = Θ^(-2i/d_k)"]
    ROT --> QR["q_m, k_n rotated"]
    QR --> DOT["q_mᵀ k_n = f(x_m, x_n, m−n)"]
    DOT --> SM["softmax → attention weights"]
    subgraph CONTEXT ["Long-context knobs (all touch θ_i or m)"]
        PI["PI: m → m·(L_train/L_new)"]
        NTK["NTK: Θ → larger"]
        YARN["YaRN: NTK-by-parts + temperature"]
    end
    CONTEXT -.-> ROT

3. Implementation

Two implementations: the dense, readable one (so you can see the math), then the version you actually ship.

# torch==2.4.0
import torch


def sinusoidal_pe(n: int, d: int, base: float = 10000.0) -> torch.Tensor:
    """(n, d) sinusoidal encoding. Note it is ADDED to the embedding."""
    assert d % 2 == 0
    i = torch.arange(0, d // 2, dtype=torch.float32)        # frequency index
    theta = base ** (-2.0 * i / d)                          # (d/2,) frequencies
    pos = torch.arange(n, dtype=torch.float32)              # (n,)
    angles = pos[:, None] * theta[None, :]                  # (n, d/2)
    # interleave sin/cos so dimension pair (2i, 2i+1) shares one frequency
    pe = torch.stack([angles.sin(), angles.cos()], dim=-1)  # (n, d/2, 2)
    return pe.reshape(n, d)


def rope_tables(seq_len: int, d_k: int, base: float = 10000.0):
    """cos/sin tables for RoPE, in the half-split (HF) convention."""
    assert d_k % 2 == 0
    inv_freq = base ** (-torch.arange(0, d_k, 2, dtype=torch.float32) / d_k)  # (d_k/2,)
    t = torch.arange(seq_len, dtype=torch.float32)
    freqs = torch.outer(t, inv_freq)              # (seq_len, d_k/2)
    emb = torch.cat([freqs, freqs], dim=-1)       # (seq_len, d_k) — repeated twice
    return emb.cos(), emb.sin()


def rotate_half(x: torch.Tensor) -> torch.Tensor:
    x1, x2 = x.chunk(2, dim=-1)
    return torch.cat([-x2, x1], dim=-1)


def apply_rope(q, k, cos, sin):
    """q, k: (B, H, T, d_k); cos, sin: (T, d_k). Rotates q and k only — never v."""
    cos = cos[None, None, :, :]     # broadcast over batch and head
    sin = sin[None, None, :, :]
    q_out = q * cos + rotate_half(q) * sin
    k_out = k * cos + rotate_half(k) * sin
    return q_out, k_out


if __name__ == "__main__":
    torch.manual_seed(0)
    d_k, total = 64, 32

    # Check 1: a rotation must preserve norms.
    q = torch.randn(1, 4, total, d_k)
    cos, sin = rope_tables(total, d_k)
    qr, _ = apply_rope(q, q.clone(), cos, sin)
    assert torch.allclose(qr.norm(dim=-1), q.norm(dim=-1), atol=1e-5)

    # Check 2: the score must depend on the OFFSET (m - n), not on absolute m.
    # Use the SAME q and k for every pair so only the rotation differs.
    q0 = torch.randn(1, 1, 1, d_k)
    k0 = torch.randn(1, 1, 1, d_k)
    cos, sin = rope_tables(128, d_k)

    def score(m: int, n: int) -> float:
        qm, _ = apply_rope(q0, q0, cos[m:m + 1], sin[m:m + 1])
        _, kn = apply_rope(k0, k0, cos[n:n + 1], sin[n:n + 1])
        return (qm @ kn.transpose(-1, -2)).item()

    print(score(0, 0), score(7, 7))                      # offset 0 pair: identical to fp error
    print(score(0, 7), score(100, 107), score(7, 114))   # offset +7 triple: also identical

Those prints are the whole theorem. The score depends only on the offset $n-m$: score(0, 0) and score(7, 7) share offset 0 and must be equal, and score(0, 7), score(100, 107) and score(7, 114) all share offset +7 and must equal each other. What must not match is the offset-0 value against the offset-+7 value. If two scores that share an offset differ by more than floating-point noise, you have a pairing or sign error — check that before suspecting the model.

It is worth running that check once even if you use a library, because the failure mode is silent and library conventions genuinely differ (see §5).

The version you ship. Every mainstream stack already has this, and you should never hand-roll it in production:

• HuggingFace transformers — LlamaRotaryEmbedding (per head, applied in LlamaAttention.forward), configured by rope_theta and rope_scaling in config.json. [as of 2026-02] the RoPE config was being consolidated under a rope_parameters key, and checkpoints written by one major version do not always load correctly in an older one — pin the transformers version alongside any checkpoint you ship.
• torchtune — exposes rope_base directly: llama3_8b() is built with rope_base=500_000, num_kv_heads=8, intermediate_dim=14336 (https://meta-pytorch.org/torchtune/main/_modules/torchtune/models/llama3/_model_builders.html).
• vLLM / SGLang / TensorRT-LLM — these must be given the same RoPE configuration as the checkpoint. A mismatch here produces fluent but subtly-wrong output rather than an error.

The knobs that matter, with realistic ranges:

4. Numbers and trade-offs

Parameter and activation overhead

For $d=4096$ and a 128k vocabulary:

The activation row is not trivial at long context. An additive encoding needs an $n\times d$ tensor; at $n=100,000$, $d=4096$, bf16, that is 800 MB per sequence — on top of the KV cache, and only at the input. RoPE adds nothing, which is one reason it is the default even before you get to quality.

Wavelengths, derived

The fastest pair in a head of $d_k=128$ has period $2\pi /{\theta }_0=2\pi /1\approx 6.28$ tokens. The slowest has $2\pi /{\theta }_{63}=2\pi {\Theta }^{(d_k-2)/d_k}$. With $\Theta =10^4$ that is $\approx 5.4\times 10^4$ tokens; with $\Theta =5\times 10^5$ it is $\approx 2.6\times 10^6$ tokens. (Asymptotically the slowest period is $2\pi \Theta$; the $(d_k-2)/d_k$ exponent is why the exact figure is a little smaller, and it is the exact figure that governs behaviour.)

That calculation is the whole long-context story. With $\Theta =10^4$, the slowest dimension completes one revolution in ~54k tokens — so a 128k-token context asks the model to distinguish positions using an angle that has already wrapped around more than twice, and any two positions separated by exactly one wavelength are indistinguishable by that dimension alone. Raising $\Theta$ to $5\times 10^5$ moves the slowest period to ~2.6M tokens and keeps the whole 128k range inside a single revolution (0.05 of one, in fact). The cost: the faster dimensions have longer periods too, which is why naive base scaling blurs local resolution and why NTK-aware/YaRN schemes scale the frequencies unevenly instead.

Attention score magnitude is unchanged

A rotation preserves norms: $\Vert R_{m}q\Vert =\Vert q\Vert$. So the variance argument behind the $1/\sqrt{d_k}$ scaling in Attention is unaffected — RoPE does not change the score distribution’s scale and does not interact with that constant. This is a real advantage over relative-bias schemes, which do add to the logits and therefore change their effective temperature (T5-style biases are typically initialised so this is small, but it is a thing to check).

Which axis you are optimizing

5. Pitfalls

Applying position after the first block (or not at all). Symptom: the model trains and reaches a plausible loss, but any task requiring order — following multi-step instructions, code, arithmetic — is at chance, and the loss curve looks suspiciously similar to a bag-of-words baseline. Diagnose by permuting the input tokens and checking the logits change; if they do not, your position signal isn’t reaching attention.

Rotating $v$ as well as $q$ and $k$. Symptom: the model trains but converges to a strictly worse loss, with no error and no shape mismatch. The output of attention is a weighted sum of values; rotating values by their absolute position injects a position-dependent transform into the output that no subsequent $W_O$ can undo (it varies per position). RoPE goes on $q$ and $k$ only — always.

Mismatched pairing convention between training and inference. Symptom: the model generates fluent text that is subtly wrong — right topic, wrong facts, degraded instruction-following — and perplexity is a few points up rather than catastrophically broken. This is the classic half-split-vs-interleaved bug and its signature is that almost everything works. The one-line test is the offset-invariance check in §3.

Caching unrotated keys. Symptom: generation degrades as the sequence grows, and prefix caching gives different results from no caching. The KV cache must store $R_n{k}_n$ — the already-rotated key at its own position — not the raw $k_n$. If you rotate at read time, you must rotate by the position the key occupied, not by the current one.

Adding both RoPE and an additive positional embedding. Symptom: slow convergence and a model that is oddly sensitive to sequence length; you have made position information twice as strong and partly redundant, and the optimizer spends capacity separating the two signals. Check config.json/model code — some models genuinely use both, but doing it by accident is common when porting a BERT-family training script to a LLaMA-family checkpoint.

Mismatched rope_scaling between the checkpoint and the server. Symptom: output quality depends on the serving framework. The same model served by transformers and by vLLM at 32k context gives measurably different answers, with no error in either log. Always read the RoPE config out of the checkpoint and pass it explicitly.

Assuming a large max_position_embeddings means the model works there. Symptom: needle-in-a-haystack passes but multi-hop and aggregation questions at the same length fail badly. The window fits, but the effective context is much shorter. Measuring this is the topic of §7.

Sinusoidal encodings with a learned base. Symptom: training-time loss looks normal but quality collapses at lengths beyond what you trained on, worse than a properly configured RoPE model. The frequencies are constants in the original design; treating $10000$ as a hyperparameter turns it into an untuned long-context intervention without the rest of the recipe.

6. Using it in a project

In 2026 you almost never choose a positional encoding. A checkpoint comes with one burned in, and if it is RoPE you have exactly one real decision: the context window, and whether the checkpoint’s RoPE config matches how you are serving it.

Reach for a positional-encoding decision when:

• You are fine-tuning a model to a longer context than it was trained on. This is the common case, and the answer is a RoPE intervention (YaRN or NTK scaling) plus a modest amount of long-context fine-tuning data — usually far less than continued pretraining would need.
• You are training from scratch, in which case RoPE with a base matched to your target context is the default and there is no serious competitor. [as of 2026-02]
• You are serving a checkpoint through a different stack than it was trained in, and need to verify the RoPE configuration survives the port.
• You need a hard context ceiling you can reason about: a learned absolute table gives you an exact wall at $n_{max}$; RoPE gives you graceful degradation whose shape you must measure.

When NOT to touch any of this:

• When your task fits in the model’s native window. Almost all production RAG and chat traffic fits in 8k–32k. Adding YaRN scaling to squeeze out 128k context, when 95% of your requests are under 4k and the long ones are summarization, buys quality loss on the common case for a capability the rare case does not need.
• When the real problem is retrieval. If the answer isn’t in the context, a larger context window does not help. Measure retrieval recall@k first — see RAG and Retrieval.
• When a streaming/multi-turn design would work better. Repositioning or re-summarizing a conversation is often cheaper and more reliable than opening the window further.
• When the model is not RoPE-based. Bidirectional encoders used for retrieval embeddings often use learned or sinusoidal absolute positions plus a fixed max_position_embeddings; the RoPE playbook simply does not apply, and forcing a RoPE checkpoint into that role loses the properties you were relying on.

The one transferable mental model. Position is a property of the attention score, not of the token vector. Adding to the residual stream is a workaround for not having found a way to put position in the score. That makes the RoPE derivation obvious and the failure modes predictable: anything added to the residual stream can be learned around; anything that changes $q\cdot k$ cannot.

7. How you would prove it worked

Positional encoding has one direct metric and one strong diagnostic.

Per-position loss. Compute the model’s loss bucketed by token position on a held-out set. Plot position on the x-axis. Two things to look for:

• Within the trained range, the curve should be roughly flat. A curve that rises steadily with position inside the training window indicates your position signal is decaying or your attention is diluting — not a long-context problem at all.
• Beyond the trained range, the curve tells you the extrapolation quality. With a learned absolute table it turns into a wall (the embeddings do not exist). With RoPE at a mismatched rope_scaling, it rises smoothly and steeply. With a correctly configured extension, it stays flat and then degrades.

The permutation test. (example) Take 200 evaluation examples, shuffle each input’s tokens at random, and measure the drop in task accuracy. A model with a working positional encoding should lose most or all of its accuracy on order-sensitive tasks; a model that barely notices has a broken or absent position signal. Report the drop, not the absolute number — the absolute number depends on the task.

The long-context ablation, done honestly. Needle-in-a-haystack saturates on almost every model that advertises a large window and therefore proves very little. Use an aggregation task instead: place k items at random positions and ask for a count or a sum. (example) A model can pass 128k needle retrieval and fall to near-chance on aggregation at 16k. Report the length at which that metric drops below its short-context baseline; that is the number to put in front of a stakeholder, not the advertised window.

Reporting template (example — replace with your own numbers). “At the checkpoint’s native 8k window we saw flat per-position loss. Extending to 32k with YaRN at factor 4, no fine-tuning, raised per-position loss beyond 8k by 0.9 nats (example) and dropped aggregation accuracy from 91% to 44% (example). After 400 steps of long-context SFT on 8k–32k sequences, loss was within 0.05 nats of the short-context baseline and aggregation recovered to 87% (example). We kept the native config for all traffic under 8k, which is 96% of requests.”

8. Interview drill

“Why does a Transformer need positional encoding at all?” Strong answer: self-attention is permutation-equivariant — the output is a weighted sum over positions with weights that depend only on $q\cdot k$ content, so reordering the input reorders the output identically. Without a position signal the model cannot distinguish “dog bites man” from “man bites dog”, and it will happily train to a plausible loss while failing every order-sensitive task. Common wrong answer: “because attention has no concept of distance.” Attention has no concept of anything positional, including order — saying “distance” suggests the problem is only long-range, when it is already broken at two tokens.

“Walk me through the sinusoidal encoding.” Strong answer: for each position $pos$ and each dimension pair $i$, emit $\sin$ and $\cos$ of $pos/10000^{2i/d}$. The $d/2$ frequencies form a geometric progression from $2\pi$ to $10000\cdot 2\pi$, giving a multi-scale clock. The reason for the specific pairing is that for a fixed offset $k$, $\mathrm{PE}(pos+k)$ is a linear function of $\mathrm{PE}(pos)$, so relative offsets are linearly recoverable. Follow-up: “so why did it lose?” → it is added to the residual stream, so position and content are entangled in the same vector from layer 0 onward, every weight has to learn to separate them, and it does not extrapolate well because the model has never seen long-range position values.

“Explain RoPE to me as if I have 90 seconds.” Strong answer: pair the head dimensions and rotate each pair by an angle proportional to position. Because $R_m^{\top }{R}_n=R_{n-m}$, the query–key inner product depends only on the relative offset, so attention becomes a function of $m-n$ while the implementation still uses absolute angles. Zero parameters, zero added memory, no change to attention’s form, and the base $\Theta$ is a single knob for context extension. Common wrong answer: “it adds a rotary positional vector to the embedding.” RoPE adds nothing to the embedding — it multiplies $q$ and $k$ after the projection, per head.

“Why is RoPE applied to $q$ and $k$ but not $v$?” Strong answer: the position information needs to affect the attention weights, which are the $q\cdot k$ products. The output is a weighted sum of values, and rotating a value would apply a position-dependent linear transform to the output that no downstream weight can invert — it injects noise, not signal. Follow-up: “so how does position reach the output?” → through which values get weighted, i.e. through the softmax distribution, which is exactly where you want it.

“I trained a 2k-context model and I need 32k. What do you do?” Strong answer: raise the effective context with a RoPE intervention — YaRN or NTK-aware scaling, which touch the frequencies rather than the positions — then run a short fine-tuning pass with long sequences in the data. Measure per-position loss and an aggregation task at 32k, not needle retrieval. Common wrong answer: “increase max_position_embeddings and the RoPE base.” Changing the base without matching training degrades quality within the original window too, and declaring a bigger window in config does not create the capability — you get fluent, confident, badly-positioned output.

“ALiBi vs RoPE — why did RoPE win?” Strong answer: both give relative position without per-offset parameters, and ALiBi extrapolates well. RoPE won on ecosystem gravity plus two concrete properties: it composes equally well with the KV cache and with FlashAttention (a rotation of $q$ and $k$ composes with any kernel that is already doing a matmul), and its frequency structure gives a natural, tunable decay and a clean long-context extension story. ALiBi’s bias has to be added to the logits, which some fused kernels handle less flexibly. Common wrong answer: “RoPE is more accurate.” On many benchmarks ALiBi is competitive; the decisive factors are engineering and extendability, and saying “more accurate” invites a follow-up you cannot answer.

“Does RoPE change the $1/\sqrt{d_k}$ scaling?” Strong answer: no. Rotations preserve norms, so the score variance is unchanged and the same constant applies. Common wrong answer: “yes, you have to rescale by the base.” That confusion comes from YaRN, which adds a temperature correction on top of RoPE as part of its extension recipe — that is a property of YaRN, not of RoPE.

Next: Modern Architecture Deltas — what changed between the 2017 paper and a 2026 model: pre-norm, RMSNorm, SwiGLU, GQA, FlashAttention, MoE, and the cost each delta was paying down.