RAG Evaluation

TL;DR — A RAG system has two failure surfaces with different metrics, and conflating them is why teams tune prompts for weeks while the retriever quietly misses the answer. Retrieval is measured with recall@$k$ (the ceiling: did the evidence make the candidate list?), MRR (how high is the first relevant hit?) and nDCG@$k$ (is the whole ordering right, with graded credit and a position discount?). Generation is measured with faithfulness (what fraction of the answer’s atomic claims are supported by the retrieved context?), answer relevance and answer correctness. Both need a golden set — expert-labelled or synthetically generated and human-validated. LLM-as-a-judge scales evaluation but is biased: it prefers the first slot (position bias), longer answers (verbosity) and its own generations (self-preference), so swap positions and average, use a different judge model, and calibrate the judge against a human-labelled subset before you ship a single number.

1. Why this exists

Picture the most common failure in practice. A team ships a RAG assistant. Users say the answers are wrong. The team spends two weeks on the system prompt — “answer only from the context”, “cite your sources”, “if unsure, say you don’t know” — and accuracy moves from 61% to 63%. Nobody has measured whether the right chunk was ever retrieved.

Then someone runs a five-line experiment: for each failing question, take the chunk that does contain the answer and paste it directly into the prompt as the only context. The model answers correctly. That single test — the gold-context test — proves the generator was never the problem. Two weeks went into the wrong layer.

This chapter exists because “the answers are wrong” is not a diagnosis. The diagnosis has to separate three structurally different failures:

The second thing that makes evaluation hard is that a RAG system is a pipeline, and end-to-end accuracy is a product of its stages. If recall@10 is 0.85 and the generator answers correctly 90% of the time given the right context, end-to-end accuracy is at most $0.85\times 0.90=0.765$. Improving the generator from 0.90 to 0.95 moves end-to-end from 0.765 to 0.808, while improving recall from 0.85 to 0.95 moves it to 0.855. Without per-stage metrics you cannot tell which is the cheaper purchase — and the retrieval fix is usually cheaper.

2. How it works

2.1 Retrieval metrics, symbol by symbol

Assume a labelled set: for each query $q$ there is a set $G_q$ of relevant chunks in the corpus, and the retriever returns a ranked list $r_1,r_2,\dots$ with $r_i$ the chunk at rank $i$.

Recall@$k$ — did the evidence make the candidate list?

$$\mathrm{recall}@k=\frac{|G_q\cap \{r_1,\dots ,r_k\}|}{|G_q|}$$

• $G_q$ — the ground-truth relevant set. Its completeness is the biggest threat to this metric: labelling only the chunks you happened to find measures recall against an incomplete denominator and inflates it. Build $G_q$ by pooling the top-50 from several retrievers and having a human mark all of them — the standard pooling assumption of IR evaluation.
• $k$ — the candidate depth. Recall is monotone non-decreasing in $k$, so recall@10 and recall@100 are not comparable; always report the $k$ you feed downstream.
• Why this is the headline retrieval metric: it is the ceiling on answerability. A reranker, a better prompt and a better generator all operate on the retrieved set, so if the answer is not in it, nothing downstream can recover it.

Precision@$k$ — $\frac{|G_q\cap \{r_1,\dots ,r_k\}|}{k}$. Fixes nothing that recall does not, but it is what determines your context cost: every non-relevant chunk at rank $\le k$ costs tokens and adds a distractor for the generator (see the lost-in-the-middle effect in The RAG Pipeline). A pipeline with recall@10 of 1.0 and precision@10 of 0.2 is retrieving the answer plus nine distractors.

MRR — how high is the first relevant chunk?

$$\mathrm{MRR}=\frac{1}{|Q|}\sum _{q\in Q}\frac{1}{{\mathrm{rank}}_q}$$

• $Q$ — the evaluation query set; $|Q|$ is its size. MRR is a mean over queries, not over documents, so every query contributes equally regardless of how many relevant chunks it has.
• ${\mathrm{rank}}_q$ — the 1-based rank of the first relevant document for $q$; the term is $0$ if nothing relevant was retrieved.
• $1/{\mathrm{rank}}_q$ — the per-query score. The reciprocal is what makes the metric a geometric penalty: rank 1 scores 1.0, rank 2 scores 0.5, rank 3 scores 0.33, rank 10 scores 0.1. Moving the answer from rank 5 to rank 1 is worth $1-0.2=0.8$; moving it from rank 20 to rank 10 is worth $0.1-0.05=0.05$. That is the right shape when the user reads the top result and stops.
• What it hides: MRR is blind to everything after the first hit. A system that retrieves one relevant chunk at rank 1 and nothing else scores a perfect 1.0 on a query where the answer needs three chunks. Use MRR for “find the one document” tasks (FAQ lookup, error-to-fix mapping) and not for synthesis tasks.

nDCG@$k$ — is the whole ordering right, with graded credit?

$$\mathrm{DCG}@k=\sum _{i=1}^k\frac{2^{re{l}_i}-1}{{\log }_2(i+1)},\mathrm{nDCG}@k=\frac{\mathrm{DCG}@k}{\mathrm{IDCG}@k}$$

• $i$ — the 1-based rank position. The sum is truncated at $k$ because you only care about the window you feed the generator.
• $re{l}_i$ — the graded relevance of the document at position $i$: for example 0 = irrelevant, 1 = mentions the topic, 2 = partially answers, 3 = fully answers. Binary labels (0/1) work too — nDCG then degenerates to a discounted hit count — but the graded case is where it earns its keep, because retrieval usually has a “this is the chunk” versus “this is a related chunk” distinction that recall cannot express.
• $2^{re{l}_i}-1$ — the gain. It is exponential in the label, so a grade-3 hit contributes $2^3-1=7$ units and a grade-1 hit contributes $1$. That is deliberate: it encodes “the user needs this document, not three vaguely related ones”. Replace it with a linear $re{l}_i$ and a query with four grade-1 hits outscores a query with one grade-3 hit, which is almost never what you meant. Remove the gain function entirely ($2^{re{l}_i}-1\to 1$ for every retrieved item) and nDCG becomes pure position weighting, i.e. it stops distinguishing relevant from irrelevant documents.
• ${\log }_2(i+1)$ — the position discount, and the reason the metric is about ranking rather than set membership. At $i=1$, ${\log }_{2}2=1$, so the top hit gets full credit. At $i=2$ the divisor is 1.585, so rank 2 keeps 63% of rank 1’s credit; at $i=5$ it keeps 39%; at $i=10$, 29%. The log decay is much gentler than the $1/i$ decay MRR implies, which is the correct choice when the generator reads the top 5 rather than the top 1 — a relevant chunk at rank 4 is genuinely useful, and a $1/i$ discount would understate that. Removing the discount entirely makes nDCG position-blind, and then it cannot tell a perfect ranking from the reversed one.
• $\mathrm{IDCG}@k$ — the DCG of the ideal ranking: sort all documents in the corpus by descending $rel$, take the top $k$, and compute DCG. This is the maximum achievable for this query at this $k$. Without dividing by it, DCG is unbounded and not comparable across queries — a query with ten relevant documents has more total gain available than one with two, so averaging raw DCGs silently weights easy queries more. Dividing by IDCG puts every query on $[0,1]$, where 1 means “the ideal ordering was achieved” and 0 means “no relevant document was retrieved”.
• Edge case you will hit: if $G_q$ is empty, $\mathrm{IDCG}@k=0$ and the ratio is undefined. Define that query’s nDCG as 0 and exclude it from the mean, or your dashboard turns red for a data problem rather than a quality problem.

MAP — the set-oriented counterpart, for tasks needing several chunks:

$$\mathrm{AP}@k=\frac{1}{\min (|G_q|,k)}\sum _{i=1}^{k}1[r_i\in G_q]\cdot \mathrm{Precision}@i,\mathrm{MAP}=\frac{1}{|Q|}\sum _q\mathrm{AP}@k$$

The indicator means only ranks where a relevant document appears contribute; each adds the precision attained at that point. The denominator normalises by how many relevant documents you could have found within $k$. The effect is that AP rewards putting all relevant documents near the top, not just the first — the right metric for comparison questions and “list all the safety warnings”.

2.2 Generation metrics

Faithfulness (also called groundedness) — is every claim in the answer supported by the retrieved context? Reference-free, which is what makes it usable in production. The computation, as implemented in RAGAS (Es et al. 2023; verified against the current source, [as of 2026-10]):

1. Decompose the response into atomic statements $c_1,\dots ,c_n$ with an LLM.
2. For each statement, run an NLI-style verdict against the concatenated retrieved contexts: supported or not.
3. Score

$$\mathrm{faithfulness}=\frac{1}{n}\sum _{j=1}^{n}1[c_j\text{ supported}]$$

• $c_j$ — an atomic statement, i.e. one factual claim. The decomposition step is where the metric is weakest: a compound sentence (“the refund window is 30 days and applies to annual plans only”) that is split into one statement instead of two lets a half-supported answer score 1.0.
• $n$ — the number of statements. The score is a ratio, so a two-sentence hedge and a twenty-claim technical answer are weighted the same. This is why a faithfulness of 0.9 means different things in different products, and why you should log $n$ alongside the score.
• $1[\cdot ]$ — the indicator, 1 if the statement is entailed by the context. Note the asymmetry: a true statement that the context does not mention counts as unfaithful. That is correct for RAG — the system is supposed to answer from the documents — but it means faithfulness is not a truthfulness metric, and a model answering from parametric knowledge gets punished.
• What it catches: hallucination and unsupported extrapolation. What it gaming looks like: “I don’t know” is perfectly faithful. Pair faithfulness with answer relevance or correctness, always.

Answer relevance — does the answer address the question? The RAGAS approach is indirect: generate $m$ questions the answer would plausibly respond to, embed them, and take the mean cosine similarity to the original question. A relevant answer produces close questions; an evasion (“I don’t know”) produces generic ones that are far away. Reference-free, and the counterweight to faithfulness.

Answer correctness — is it right against a reference answer? Needs references, so it is only available on the golden set, but it is the closest thing to the metric the user experiences.

Context precision / context recall — the same retrieval ideas computed without labelled relevant chunks, by asking an LLM whether each retrieved chunk is relevant (precision) and whether each reference claim is supported by the retrieved context (recall). Practical substitutes when you cannot label, and they inherit every judge bias in §2.3.

2.3 LLM-as-a-judge and its biases

Judging with an LLM (Zheng et al. 2023, MT-Bench and Chatbot Arena) reaches over 80% agreement with human preference — enough to make it the default for regression testing, not enough to make it truth. The biases that will actually bite you:

• Position bias. In pairwise judging, the model prefers whichever answer sits in the first slot (or in some model versions, the second). Mitigation: judge both orders — $(A,B)$ and $(B,A)$ — and average; a pair that wins both orders is a real win, a pair that wins once each is a tie. This doubles judge cost and is still the cheapest correctness fix available.
• Verbosity bias. Longer answers are preferred even when they add nothing. Mitigation: length-match the arms you compare, or include an explicit “length is not a quality signal” clause in the rubric, or report a length-controlled score. In RAG specifically, verbosity bias means a retriever that returns more context can look better purely because the answer got longer.
• Self-preference (self-enhancement). A judge favours text produced by its own family (Panickssery et al. 2024). Mitigation: never let the judge be the generator when you are comparing a system against itself — use a different model family, or a jury of two models from different families and report disagreement.
• Sycophancy and rubric drift. The judge agrees with any stated “reference answer” it is shown, and small rubric wording changes move scores by several points. Mitigation: judge against the criterion, not the reference verdict, and freeze the rubric in version control — a rubric edit is a metric change that invalidates every historical number.

The rule that matters. Do not ship an evaluation whose judge you have not validated against humans. Label 50–100 examples by hand, measure the judge’s agreement on that subset, and report the judge’s score as relative and directional. If your judge agrees with humans 65% of the time, it cannot resolve a 3-point difference between two systems.

2.4 The diagnostic order

flowchart TD
    A["end-to-end accuracy is bad"] --> B["recall@k on golden set"]
    B -->|low| R["retrieval problem: chunking, embedder, hybrid, query rewrite"]
    B -->|ok| C["gold-context test"]
    C -->|model answers| R2["retrieval problem: the gold chunks were not retrieved"]
    C -->|model still wrong| D{"does the fact exist in the corpus?"}
    D -->|no| K["knowledge gap: add data, or abstain"]
    D -->|yes| G["generation problem: prompt, model, context formatting"]

Run it in that order every time. It is cheap and it prevents the most expensive mistake in this domain: tuning the generator when retrieval is the bottleneck.

3. Implementation

Three pieces: retrieval metrics you can read and trust (pure Python, no dependencies), RAGAS for the LLM-judged generation metrics, and a hand-written pairwise judge that does the position swap.

"""Retrieval metrics from the definitions in §2.1. No dependencies."""
from __future__ import annotations

import math
from collections.abc import Iterable, Sequence


def recall_at_k(ranked: Sequence[str], relevant: Iterable[str], k: int) -> float:
    """|relevant ∩ retrieved@k| / |relevant|. The denominator is the *labelled*
    relevant set, so an incomplete label set inflates this."""
    rel = set(relevant)
    if not rel:
        raise ValueError("query has no labelled relevant documents")
    return len(rel & set(ranked[:k])) / len(rel)


def precision_at_k(ranked: Sequence[str], relevant: Iterable[str], k: int) -> float:
    return len(set(relevant) & set(ranked[:k])) / k


def reciprocal_rank(ranked: Sequence[str], relevant: Iterable[str]) -> float:
    """1 / rank of the first relevant item; 0.0 if none is retrieved."""
    rel = set(relevant)
    for rank, doc_id in enumerate(ranked, start=1):     # 1-based
        if doc_id in rel:
            return 1.0 / rank
    return 0.0


def dcg_at_k(grades: Sequence[float], k: int) -> float:
    """graded relevance at each rank, best-first: (2^rel - 1) / log2(i + 1)."""
    return sum((2.0 ** g - 1.0) / math.log2(i + 1)
               for i, g in enumerate(grades[:k], start=1))


def ndcg_at_k(retrieved_grades: Sequence[float],
              all_labelled_grades: Sequence[float], k: int) -> float:
    """IDCG must come from the ideal ordering of ALL labelled documents —
    including relevant ones the retriever never returned. See the note below."""
    idcg = dcg_at_k(sorted(all_labelled_grades, reverse=True), k)
    if idcg == 0.0:
        return 0.0                       # no relevant docs: define the score as 0
    return dcg_at_k(retrieved_grades, k) / idcg

Two things about this code. IDCG takes the full label vector, not your retrieved list — sorting only what the retriever returned measures the best ordering of your own output, inflates the score and hides recall failures. And k must be the same everywhere: a recall@100 headline with a generator reading the top 5 is how a pipeline reports good numbers and bad answers.

Now the generation metrics, using RAGAS. The API changed materially between the 0.1 and 0.4 lines: this is the ragas.metrics.collections API with an async, instructor-backed LLM, verified against the source tree at ragas==0.4.3 [as of 2026-10].

"""
LLM-judged generation metrics. ragas==0.4.3 [as of 2026-10]:
    pip install "ragas==0.4.3" "openai>=1.0"

The 0.4 line moved to instructor-based structured output and async scoring; the
old `from ragas.metrics import faithfulness` shim is deprecated, and metric
prompts/definitions have changed across releases. Report the version with any
number you publish.
"""
import asyncio
from openai import AsyncOpenAI
from ragas.llms.base import llm_factory
from ragas.metrics.collections import Faithfulness

judge_llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())  # NOT the generator
faithfulness = Faithfulness(llm=judge_llm)                    # 2 LLM calls/sample

SAMPLE = {"user_input": "What is the refund window for annual plans?",
          "response": "Annual plans can be refunded within 30 days of purchase.",
          "retrieved_contexts": ["Annual subscriptions are refundable within 30 days."]}


async def score_one(sample: dict) -> float:
    result = await faithfulness.ascore(**sample)
    return result.value      # statements supported / statements generated


print(asyncio.run(score_one(SAMPLE)))   # 1.0

Note what the metric does internally, from the source: generate atomic statements, run an NLI verdict per statement against the joined contexts, return faithful / total. The failure modes are inherited from those two LLM calls, not from the ratio.

Finally, the pairwise judge with the position swap — the pattern to write yourself, because this is where the biases live:

# Pairwise grading with position-swap. The swap is what neutralises position
# bias, and using a judge from a different family than the generator is what
# neutralises self-preference.
JUDGE_PROMPT = """\
Judge ONLY whether the response is supported by the context and answers the
question.

Rules:
- Length is not a quality signal. A shorter correct answer beats a longer one.
- Do not reward confident phrasing. Do not use outside knowledge.
- If the response contains any claim not supported by the context, mark it
  unsupported even if the claim is true.

Question: {question}
Retrieved context: {context}
Response: {response}

Reply with exactly one word: SUPPORTED or UNSUPPORTED.
"""


def judge_pair(llm, question: str, context: str, a: str, b: str) -> str:
    """Return 'a', 'b', or 'tie', consistent under both position orders."""
    def verdict(first: str, second: str) -> str:
        def one(resp: str) -> str:
            return llm(JUDGE_PROMPT.format(question=question, context=context,
                                           response=resp)).strip().upper()
        pa, pb = one(first), one(second)
        if pa == pb:
            return "tie"
        return "a" if pa == "SUPPORTED" else "b"

    forward, backward = verdict(a, b), verdict(b, a)   # a then b in slot 1
    # `backward` names its winner in the swapped frame, so flip its labels back
    # into the forward frame -- "a" there is `b` here, and vice versa -- before
    # comparing. Without this both calls return "a" for the same winner and every
    # consistent verdict looks like a tie.
    backward = {"a": "b", "b": "a"}.get(backward, backward)
    return forward if forward == backward else "tie"

The knobs and their realistic values:

The judge prompt is deliberately binary: ordinal scales (1–5) invite rubric drift, whereas SUPPORTED/UNSUPPORTED is a decision a human can make reliably too.

4. Numbers and trade-offs

How much data do you need?

This is the question that decides whether your evaluation is worth anything, and it is answerable. For a single accuracy estimate $\hat{p}$ over $n$ independent questions, the standard error is $\sqrt{\hat{p}(1-\hat{p})/n}$, so the 95% confidence interval half-width is

$$\epsilon =1.96\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$$

At $\hat{p}=0.7$:

The design change is pairing. Both systems run on the same questions, so the variance that matters is that of the difference, governed by how often the two disagree. With discordant fraction $d$ the paired standard error is roughly $\sqrt{d/n}$: at $d=0.2$, $n=200$ that is 0.032, i.e. ±6 points at 95%. Treating the arms as independent, the 95% interval on the difference has half-width $1.96\sqrt{2\hat{p}(1-\hat{p})/n}=1.96\sqrt{2\times 0.7\times 0.3/200}\approx 0.090$, i.e. ±9 points — so pairing narrows the interval by $9.0/6.2=1.45\times$. Pairing is free; use it.

What evaluation costs

LLM-judged metrics cost generations. Faithfulness is two calls per sample (statement generation, then NLI verdict). A 500-sample, 5-metric suite with a swap-enabled pairwise judge is thousands of judge calls per run (example: 500 × (2 + 1 + 2 + 2) ≈ 3,500) — a few dollars, cheap enough to run per pull request, expensive enough that you should cache by content hash. Human labelling is the real budget line: a graded label takes minutes, so a 300-question golden set is 15–30 expert hours.

5. Pitfalls

Position and verbosity bias. Symptom: the arm you evaluated first (or second) wins consistently and swapping the order flips the result; or the arm that returns more context wins although per-claim faithfulness is unchanged, and answers get steadily longer. Fix: judge both orders and count only consistent wins; length-match the arms and report answer length beside every quality score.

Self-preference and an unvalidated judge. Symptom: the new model wins 70% of comparisons when the judge is the new model and 52% under a different-family judge; or judge scores improve for six releases while complaints do not move and nobody can say what a 0.83 means. Fix: a judge from a different family (or a two-model jury with the disagreement rate reported), plus 50–100 hand labels to measure agreement. If agreement is 65%, treat differences under ~10 points as unmeasurable.

A golden set built entirely by an LLM, with an incomplete relevant set. Sampling a chunk and asking a model to write a question only that chunk answers is cheap and effective, but the questions are biased toward the generator’s phrasings and toward chunks that are easy to write questions about — definitions, tables, self-contained paragraphs. And if you then label only the chunks you happened to find, recall is measured against an incomplete denominator. Symptom: recall@10 of 0.95 offline while users report misses, with failures concentrated in procedural or multi-clause queries and questions whose answer is split across chunks. Fix: pool candidates from several retrievers before labelling, label with a budget, validate a random sample by hand, and inject real user queries plus deliberately hard multi-hop ones.

Gaming faithfulness with abstention. Symptom: faithfulness rises after you add “if unsure, say you don’t know” — because “I don’t know” has zero unsupported claims. Fix: always pair faithfulness with answer relevance or correctness; a system that abstains on everything is perfectly faithful and useless.

Metric shopping. Symptom: the report quotes whichever metric moved (MRR in the run where recall fell), and a 6-point offline improvement does not reproduce in production. Fix: fix the primary metric, $k$ and golden-set version before the experiment, and hold out a frozen test split.

Tuning generation when retrieval is the bottleneck. Symptom: two weeks of prompt work moves accuracy ±2 points, the gold-context test shows the generator was fine, and nobody has ever computed recall@$k$. Fix: the diagnostic order in §2.4.

6. Using it in a project

Build a golden set when the system will change more than once, or when someone has to justify a release. 100–300 questions with graded relevance labels is enough for directional paired comparisons, provided they came from real traffic and cover your query classes.

Do not build one when the corpus is small enough to eyeball, when the product is pre-iteration and you are still discovering what the questions are, or when the decision on the table is categorical. Run the gold-context test on 20 failing cases instead: an hour, and it tells you which layer to fix.

Cheap proxies before the golden set exists: log every query with its retrieved chunk ids; read twenty a week; track the fraction of answers that cite a source; and count “I don’t know” responses, because a sudden rise is usually a retrieval regression wearing a safety costume.

When NOT to pay for LLM judges: regression testing of a deterministic component — parser output, citation formatting, tool-call arguments — wants exact assertions.

The budget split that works. Most teams over-invest in generation metrics and under-invest in retrieval metrics, because generation metrics are easier to produce and prettier to look at. Invert it: retrieval metrics are cheap, they explain most end-to-end failures, and they do not require a judge you have to trust.

7. How you would prove it worked

Fix the metric before the change. Write down the primary metric, the $k$, the golden-set version and the judge model in the experiment plan. If the metric changes after the result is known, the result is not evidence.

Report the pipeline per stage, so attribution is visible:

The story this table tells — retrieval improved, generation unchanged and not the bottleneck — is the reason to build it. A single end-to-end number cannot distinguish that from a generation improvement.

Ship a regression set, not a benchmark. Freeze the golden set, run it in CI on every change that touches retrieval, chunking, the prompt or the model, and fail the build on a statistically meaningful drop. Because runs are paired, a 5-point drop on 200 questions is detectable and a 2-point drop is not — do not gate on the latter.

Calibrate the judge once and record it. Hand-label 50–100 examples, compute agreement, and store that number next to the judge’s name and version so every later score carries a visible uncertainty budget.

8. Interview drill

“How do you know a change helped?” Strong answer: a fixed primary metric on a frozen golden set, a baseline measured the same way, a paired comparison on the same questions, and an interval rather than a point — then per-stage attribution, so a retrieval fix is not credited to the prompt. Common wrong answer: “the answers look better”, or a single end-to-end number with no baseline or interval.

“Explain nDCG to me.” Strong answer: the gain $2^{re{l}_i}-1$ rewards graded relevance exponentially so a fully answering chunk outweighs several partially relevant ones; the discount ${\log }_2(i+1)$ encodes that position matters; dividing by IDCG — the DCG of the ideal ordering, computed from all labelled documents — makes the score comparable across queries. Common wrong answer: calling it “recall with weights”, or building IDCG from the retrieved list only.

“An LLM judge says the new system is better. What do you check?” Strong answer: position bias first (re-run with the arms swapped — a result that flips is not a result), then verbosity bias (compare answer lengths), then self-preference (is the judge the same family as the generator?), then the judge’s agreement with a human-labelled subset. Only a difference surviving all four is reportable. Common wrong answer: trusting the judge because it agrees with you on the examples you eyeballed.

“Your retriever recall@10 is 0.92 and users still say the answers are wrong. What now?” Strong answer: the gold-context test on the failing cases, then a search of the raw corpus for a distinctive string from the answer. Three outcomes: the chunk was retrieved but the generator ignored it (context formatting, ordering, or lost-in-the-middle — put the relevant chunk first); the fact is not in the corpus at all (knowledge gap — no retriever change helps); or the labelled relevant set was incomplete and recall@10 was never really 0.92. Common wrong answer: swapping the embedding model, which moves none of those three.

“How large does your eval set need to be?” Strong answer: it depends on the effect you want to detect. The 95% interval half-width on a single accuracy near 0.7 is $1.96\sqrt{0.21/n}$ — about ±9 points at $n=100$, ±6.4 at $n=200$. Paired comparison on the same questions is roughly 1.45× as sensitive, because only the discordant pairs carry variance. Common wrong answer: “a few dozen is fine” — that resolves nothing smaller than a double-digit change.

Next: Intent Recognition and Routing — evaluation tells you that a pipeline is failing; the next chapter is about sending each query to the pipeline that can actually answer it.