The RAG Pipeline

TL;DR — RAG is two pipelines, not one: an offline indexing pass (parse → chunk → embed → index) and an online serving pass (transform → retrieve → rerank → assemble → generate → cite). Most disappointing RAG systems fail in the middle of the serving pass — retrieval — but get debugged at the end, because a wrong answer looks like a generation problem. The decisive, cheap experiment is the gold-context test: hand the generator the chunk that actually contains the answer. If it then answers correctly, your retriever is the bug, and no amount of prompt engineering will fix it. Measure recall@k before you touch a prompt.

1. Why this exists

A team ships an internal docs assistant over 12,000 pages of engineering docs. The demo works. In production, users report that it confidently states things that are wrong, and cites a page that does not say that.

The team’s instinct is that the model is hallucinating. So they upgrade the generator, add “Only answer from the context above. If the context does not contain the answer, say you do not know,” and add a chain-of-thought step. The answers get more fluent and no more correct.

The actual bug, in this pattern, is almost always upstream: the page that answers the question ranks 74th by cosine similarity, and the system retrieves the top 5. The generator never saw the evidence, and a well-prompted model with no evidence produces a plausible answer — that is what it is trained to do. The prompt was never the problem.

This is the single most important fact about RAG in practice:

Most disappointing RAG systems are retrieval failures that were misdiagnosed as generation failures, because nobody measured retrieval.

Why the misdiagnosis is so persistent: the generation stage produces an artifact you can read (the answer), while the retrieval stage produces a list of chunk IDs you have to be instrumented to see. Reading the answer is free; measuring recall@k requires a labelled set. So teams optimise the thing they can see.

RAG exists because of a structural property of language models: parametric knowledge is stale (frozen at the training cut-off), unverifiable (you cannot cite a weight), unauditable (you cannot delete one fact), and unattributable (the model cannot tell you where it learned something). RAG moves the knowledge out of the weights and into a store you control — one you can update in seconds, cite precisely, and filter per user. The original formulation is Lewis et al. 2020, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401, which trained a DPR retriever and a BART generator jointly. Almost nothing you will build in 2026 looks like that paper — it is the concept that survived, not the architecture.

2. How it works

2.1 Two passes with different failure economics

flowchart TB
    subgraph OFFLINE["Offline — indexing (batch, minutes to hours)"]
        D["Documents<br/>(PDF, HTML, MD, tickets)"] --> P["Parse<br/>to structured text + metadata"]
        P --> C["Chunk<br/>to N passages"]
        C --> E["Embed<br/>to N x d matrix"]
        E --> V["Vector index<br/>(HNSW / IVF-PQ)"]
        C --> L["Lexical index<br/>(BM25 / inverted)"]
        C --> M["Metadata store<br/>(doc id, section, ACL, version)"]
    end
    subgraph ONLINE["Online — serving (per query, tens to thousands of ms)"]
        Q["Query"] --> QT["Transform<br/>(rewrite / expand)"]
        QT --> R["Retrieve<br/>top-N from vector + lexical"]
        V --> R
        L --> R
        R --> F["Fuse<br/>(RRF)"]
        F --> RR["Rerank<br/>N to k"]
        RR --> A["Assemble<br/>context + ordering + citations"]
        M --> A
        A --> G["Generate"]
        G --> CIT["Cite / abstain / verify"]
    end

The crucial asymmetry: indexing cost is amortised over every query; retrieval cost is per query but tiny; generation cost is per query and dominant. That shapes where to spend effort. Doubling your embedding model’s size increases index build time and storage once, and adds a few milliseconds to every query. Doubling the number of chunks you stuff into the prompt doubles your generation cost forever. Optimising the index is cheap; over-stuffing the prompt is not.

2.2 The contract of each stage

A pipeline is only debuggable if each stage has a stated input, output, and invariant. These are the ones that matter:

Read that table as a list of places to look when the output is wrong, top to bottom. Most of the time, the answer is in the first three rows.

2.3 The arithmetic that decides whether your system can work at all

Three quantities govern everything, and they are worth writing down before you write any code.

The answerability ceiling. Let $Recall@k$ be the fraction of queries for which at least one gold passage appears in the retrieved top-$k$, and let $g$ be the probability the generator produces a correct, grounded answer given that the gold passage is in context. Then

$$P(\text{correct})\le Recall@k\times g$$

• $Recall@k$ — the retrieval ceiling. It is a property of the chunker, the embedder, the index, and the query transformation — not of the generator. It multiplies everything downstream, so a 0.30 recall can never yield a better-than-0.30 system no matter how good $g$ is.
• $g$ — the conditional generation quality. It is bounded above by 1, and for a modern frontier model with genuinely relevant context it is often near 1. In other words, the ceiling is almost entirely set by $Recall@k$. If recall is 0.60 and $g$ is 0.95, you get 0.57 — and the only lever that moves it is recall. Getting $g$ from 0.95 to 1.0 buys you 3 points. Getting recall from 0.60 to 0.85 buys you 24.

The inequality is loose in one direction (the generator can occasionally answer correctly from parametric knowledge alone, which is exactly the hallucination you are trying to eliminate), and it is the reason the correct first move for a weak RAG system is almost never a prompt change. It is a retrieval measurement.

The context budget. The prompt you actually send is

$$B_{\text{prompt}}=B_{\text{sys}}+\sum _{i=1}^k{t}_i+B_q+B_{\text{out}}$$

• $B_{\text{sys}}$ — the system prompt and any few-shot exemplars. Usually the most stable term and the most compressible one.
• $t_i$ — the token count of the $i$-th retrieved chunk. The sum is the term you control by choosing $k$ and the chunk size, and it is the term that dominates cost and latency.
• $B_q$ — the user query, plus any injected conversation history or rewritten query.
• $B_{\text{out}}$ — the reserved output/reasoning budget. Reserve it up front; a model that runs out of room mid-answer truncates, or silently degrades.

The cost split. Per query, the cost is

$$\text{cost}=\underset{\text{one short text}}{\underbrace{C_{\text{embed}}(1)}}+\underset{\text{sublinear}}{\underbrace{C_{\text{ANN}}(N)}}+\underset{\text{linear in }N}{\underbrace{C_{\text{rerank}}(N)}}+\underset{\text{linear in context}}{\underbrace{C_{\text{gen}}\left(\sum _i{t}_i\right)}}$$

$C_{\text{embed}}$ on a single query is one forward pass through a model two orders of magnitude smaller than the generator. $C_{\text{ANN}}$ is a graph or cluster walk. $C_{\text{rerank}}$ is $N$ cross-encoder forward passes — real, and usually the p99 latency term at $N\approx 100$ (see Retrieval: Hybrid Search and Reranking). $C_{\text{gen}}$ dominates, because it scales with every token you retrieved and because it is served by the largest model in the stack.

The practical consequence: $k$ is your main cost knob, and $N$ (the candidate depth before reranking) is nearly free by comparison. Dropping $k$ from 20 to 5 saves generation cost but caps recall. Dropping $N$ from 100 to 20 saves reranker cost but caps the reranker’s input, which is where precision comes from. Err towards a deep $N$ and a small, well-ranked $k$.

2.4 The runtime sequence

transform → embed query → ANN search top-N → fuse → rerank to k → assemble → generate → verify citations or abstain. Every technique in this part of the book — hybrid search, reranking, query rewriting, GraphRAG — intervenes at one of those arrows, and is only worth adding if you can name the arrow and measure it.

3. Implementation

A real, runnable naive RAG loop with no framework in the way. It runs end-to-end against a local embedding model and prints the retrieved chunks and the prompt; the LLM call is injected so the file has no network dependency.

# naive_rag.py — a complete naive RAG pipeline in well under 100 lines.
#
# Deps: pip install "sentence-transformers>=3.0" "numpy>=1.26"
# The SentenceTransformer / encode(normalize_embeddings=...) surface has been
# stable across the 3.x-5.x line [as of 2026-10]. Pin your own version in
# production: changing the embedding model invalidates every vector you stored.
#
# Deliberately absent: hybrid search, reranking, query rewriting, ANN indexing.
# At under ~100k chunks a brute-force dot product takes single-digit
# milliseconds and removes an entire class of recall bugs. Add HNSW when you can
# measure the recall you lose.

from __future__ import annotations

from dataclasses import dataclass

import numpy as np
from sentence_transformers import SentenceTransformer

EMBED_MODEL = "BAAI/bge-base-en-v1.5"   # 768-dim, English, 512-token max input
# BGE-v1.5 models want this prefix on QUERIES ONLY, never on documents:
# https://github.com/FlagOpen/FlagEmbedding (model list, "query instruction")
QUERY_PREFIX = "Represent this sentence for searching relevant passages: "


@dataclass
class Chunk:
    id: str
    text: str
    source: str
    start: int          # character offset in the source document


def chunk_markdown(text: str, source: str, size: int = 1200,
                   overlap: int = 200) -> list[Chunk]:
    """Character-based recursive split. See ./chunking.md for why these numbers
    are a starting point and not a policy."""
    assert 0 <= overlap < size, "overlap must be < size, or the stride is <= 0"
    stride = size - overlap
    out: list[Chunk] = []
    pos = 0
    while pos < len(text):
        window = text[pos: pos + size]
        if len(window) == size:                    # prefer a language boundary
            cut = max(window.rfind("\n\n", int(size * 0.7)), window.rfind(". "))
            if cut > 0:
                window = window[: cut + 1]
        if window.strip():
            out.append(Chunk(id=f"{source}#{pos}", text=window.strip(),
                             source=source, start=pos))
        if len(window) < stride:
            break
        pos += max(stride, len(window) - overlap)
    return out


class Retriever:
    def __init__(self, model_name: str = EMBED_MODEL) -> None:
        self.model = SentenceTransformer(model_name)
        self.chunks: list[Chunk] = []
        self.matrix: np.ndarray | None = None      # (N, d), L2-normalised

    def index(self, chunks: list[Chunk], batch_size: int = 64) -> None:
        self.chunks = chunks
        # normalize_embeddings=True means a plain dot product IS cosine
        # similarity. Do it once here rather than re-normalising per query.
        self.matrix = self.model.encode(
            [c.text for c in chunks],
            batch_size=batch_size,
            normalize_embeddings=True,
            convert_to_numpy=True,
        ).astype(np.float32)

    def search(self, query: str, k: int = 5) -> list[tuple[Chunk, float]]:
        assert self.matrix is not None, "call .index() first"
        q = self.model.encode([QUERY_PREFIX + query], normalize_embeddings=True,
                              convert_to_numpy=True)[0].astype(np.float32)
        scores = self.matrix @ q                   # (N,) cosine similarities
        top = np.argpartition(-scores, min(k, len(scores) - 1))[:k]
        top = top[np.argsort(-scores[top])]        # sort only the k we kept
        return [(self.chunks[i], float(scores[i])) for i in top]


PROMPT = """Answer the question using ONLY the numbered context below.
Cite the bracketed id of every source you rely on, like [doc.md#1200].
If the context does not contain the answer, reply exactly: NOT_IN_CONTEXT.

{context}

Question: {question}
Answer:"""


def assemble(hits: list[tuple[Chunk, float]], question: str) -> str:
    # Most relevant FIRST. Long-context models degrade on material buried in the
    # middle (Liu et al. 2023, https://arxiv.org/abs/2307.03172), and the first
    # slot is the least ambiguous place to put the best evidence.
    ctx = "\n\n".join(f"[{c.id}] (from {c.source})\n{c.text}" for c, _ in hits)
    return PROMPT.format(context=ctx, question=question)


def rag(question: str, retriever: Retriever, k: int = 5, generate=None) -> dict:
    hits = retriever.search(question, k=k)
    prompt = assemble(hits, question)
    # `generate(prompt) -> str` is your LLM call, left injectable so this file
    # needs no network. A grounded system should ALSO verify that every bracketed
    # id in the answer exists in `hits`; if not, that is a citation
    # hallucination and should be an error, not a log line.
    answer = generate(prompt) if generate else "(no generator wired in)"
    return {"prompt": prompt, "hits": hits, "answer": answer}

Four details in that code decide whether it works:

• normalize_embeddings=True on both sides, so the dot product is cosine. Normalise once at index time, not per query — see Embedding Models.
• QUERY_PREFIX on queries only. E5, BGE, GTE-Qwen and every instruction-tuned embedder are trained with a query-side prefix and rank worse without it. No error, no warning: the most common silent recall bug in RAG.
• np.argpartition then sort $k$. Top-$k$ of $N$ is $O(N)$; a full sort is $O(N\log N)$. At $N=100,000$ that is a profile line, not a rounding error.
• hits returned alongside the answer, so the same call that generates can be evaluated for retrieval. A pipeline that discards its retrieved ids is one you cannot debug.

The knobs that matter

4. Numbers and trade-offs

Where the milliseconds actually go

A per-query latency budget for a production-shaped system (example):

The reranker is the stage people forget to budget. It is $O(N)$ transformer forward passes with no batching across queries, and it is frequently the difference between “fast” and “compliance says we cannot ship it”.

Recall is the ceiling, and it decays fast with $k$

Without labels you can still reason about the shape. Dense retrieval on heterogeneous corpora is weak zero-shot: BEIR (Thakur et al. 2021, arXiv:2104.08663) benchmarked 10 retrieval systems across 18 datasets and found that BM25 is a robust baseline, while dense and sparse retrievers, though far more efficient, “often underperform other approaches” out of distribution. Rerankers and late-interaction models did best, “at high computational costs.”

That is the empirical case for the two-stage design in one sentence: the cheap retriever’s job is to get the answer into the top 100; the expensive reranker’s job is to drag it into the top 5. A pipeline that sets $N=k=5$ has thrown away the reranker’s entire job.

5. Pitfalls

Each of these has a symptom — the observable that tells you it happened. Symptoms matter more than causes, because you cannot fix what you cannot see.

Retrieval failure misread as hallucination. The gold passage ranked below $k$. Symptom: the answer is fluent, plausible, on-topic, and wrong; the citations point at topically related but non-answer-bearing chunks. Test: the gold-context test in §7. Fix the retriever, not the prompt.

Wrong prefix discipline. Retrieving with a query prefix on documents, or not on queries. E5 expects query: / passage: ; BGE-v1.5 expects “Represent this sentence for searching relevant passages: “ on queries; instruction-tuned embedders expect an instruction. Symptom: retrieval quality is mediocre and uncorrelated with model size — swapping in a bigger embedder does not help. There is no exception and no warning.

Forgetting to re-index after changing the embedding model. Embedding spaces are not comparable across models, or even across versions of one model. Symptom: recall collapses after a “harmless” dependency bump; every query returns roughly random chunks. Version the embedder in the index metadata and refuse to serve a mismatch.

Chunks larger than the embedder’s max sequence length. Anything past the limit is silently truncated. Symptom: questions whose answer lies in the tail of a long chunk are never retrieved, while questions about the head of the same chunk work fine. Compare max_seq_length against your chunk size in tokens, not characters.

A bad chunk boundary makes a correct answer uncitable. The chunk is the unit of both retrieval and citation. Symptom: the answer is right but the citation points at a chunk that does not contain the sentence. Users lose trust even when you are correct.

Stuffing the context because “more is better”. Liu et al. 2023 (Lost in the Middle, arXiv:2307.03172) found a U-shaped position curve: models attend best to the beginning and end of a long context and degrade in the middle. Symptom: adding the relevant chunk makes the answer worse than not adding it, or leaves it unchanged. Fix: fewer, better-ranked chunks, best one first.

No abstention path. Symptom: the system answers every question, including ones about products that do not exist, with total confidence. If nothing in the corpus matches, an LLM told “answer the question” will answer from its weights. Give it an explicit no-answer token and count how often it fires.

Evaluating with a judge you never validated. LLM judges carry position, verbosity, and self-preference biases (Zheng et al. 2023, arXiv:2306.05685). Symptom: your eval scores are great and your users are unhappy. Calibrate against a human-labelled subset, and treat judge scores as directional rather than absolute.

6. Using it in a project

Decide whether to build RAG before deciding how

Anthropic’s contextual-retrieval write-up states the threshold bluntly: if your knowledge base is smaller than 200,000 tokens (about 500 pages), just put the whole thing in the prompt. With prompt caching they report that approach cuts latency by more than 2× and cost by up to 90% on repeated prompts (Anthropic, 2024). This is the most commonly skipped decision in the whole discipline. RAG is infrastructure: an index, a sync job, an embedding-model version to track, an evaluation set, and a re-index pipeline. Do not pay for it when the corpus fits.

The distinction that trips people up: RAG changes what the model can see; fine-tuning changes what the model does. A support bot that needs to know this quarter’s pricing wants RAG. A support bot that must reply in your brand voice with a fixed JSON schema wants fine-tuning. A bot that needs both wants both, and that is normal.

If you do build it, build it in this order

1. Golden set first. 50–200 real questions paired with the passage that answers each. Without it you optimise blind. Synthetic questions are cheap but biased towards chunks that are easy to write questions about — validate a sample.
2. Baseline: BM25 alone. No training, no GPU, no index build; BEIR found it a robust baseline out of distribution.
3. Add dense retrieval; measure recall@100. Now you know whether embeddings earn their keep on your data.
4. Add fusion (RRF) and a reranker — see Retrieval: Hybrid Search and Reranking.
5. Then, and only then, tune the prompt and pick the generator.

Steps 1–4 are the job. Step 5 is where most teams spend 90% of their time.

7. How you would prove it worked

The gold-context test (do this first; it costs about an hour). Take 50 failing queries. For each, hand-build a prompt containing only the passage that contains the answer, and run your existing generator on it.

This one experiment partitions the entire failure space, and it is the cheapest high-signal measurement in RAG. Run it before touching anything else.

Retrieval metrics on the golden set. With relevance labels:

$$Recall@k=\frac{|\text{relevant}\cap \text{retrieved@}k|}{|\text{relevant}|},\mathrm{MRR}=\frac{1}{|Q|}\sum _q\frac{1}{{\mathrm{rank}}_q}$$

• $Recall@k$ — of all passages that could answer the query, what fraction are in the top $k$? This is the ceiling from §2.3. Report it at both $N$ (before reranking) and $k$ (after).
• ${\mathrm{rank}}_q$ — the 1-based position of the first relevant item for query $q$; MRR averages its reciprocal. MRR is the right metric when exactly one document answers the question; recall@$k$ is the right one when several do.

Report $Recall@N$ and $Recall@k$ separately. If $Recall@100=0.92$ but $Recall@10=0.55$, your retriever is fine and your reranker (or its absence) is the bug. If $Recall@100=0.55$, no reranker will save you, and every hour spent on the reranker is wasted.

End-to-end metrics. Faithfulness/groundedness (decompose the answer into atomic claims, judge each against the retrieved context), answer relevance, and context precision. RAGAS (Es et al. 2023, arXiv:2309.15217) popularised reference-free versions of these, but its metric definitions and prompts have changed across releases — report the RAGAS version alongside any number you publish. Full metric definitions are in RAG Evaluation.

Ablation table (example — replace with your own numbers). Each row changes exactly one thing against the row above, on the same golden set:

Read the table the way an interviewer would. Rows 2→3 show that raising $N$ buys recall@100 and nothing at $k$ — that is the reranker’s job. Row 4→5 is the reranker earning its 420 ms. Row 5→6 is an indexing-time change moving recall@100 from 0.89 to 0.94, which is impossible for any reranker to do. That is how you tell which layer a gain came from.

Reporting template. “We built a 200-question golden set from support tickets. Recall@100 was 0.81 with dense retrieval alone, so the generator was not the bottleneck. Adding RRF hybrid search and a cross-encoder reranker (N=100 → k=10) raised recall@10 from 0.61 to 0.86 (example) at p99 = 480 ms (example). The gold-context test showed the generator answered correctly from gold context in 47 of 50 failures, confirming retrieval as the binding constraint.”

8. Interview drill

“Our RAG system hallucinates. What do you do?” Strong answer: separate retrieval from generation before changing anything. Run the gold-context test on a sample of failures: if the model answers correctly when given the answer-bearing passage, the failure is retrieval, and the fix lives in chunking, embeddings, reranking, or query rewriting. If it still fails, it is a generation failure, and the fix is the prompt or the model. If the passage is not in the corpus at all, it is a knowledge gap: add a data source and an abstention path. Then instrument recall@$N$ and recall@$k$ separately. Common wrong answer: “improve the prompt — add ‘only answer from the context’.” That suppresses some hallucinations and cannot fix a missing passage; it usually converts a fluent wrong answer into an unhelpful refusal.

“How do you know your retriever is good enough?” Strong answer: measure recall@$N$ against a golden set, where $N$ is your retrieval depth. It is the ceiling on end-to-end accuracy. If recall@100 is below roughly 0.9 you have no business tuning the generator. Recall is the only retrieval metric that matters for feasibility; precision is what the reranker is for. Common wrong answer: quoting NDCG@10 with no golden set, or reporting a single “retrieval accuracy” number that conflates retrieval depth with ranking quality.

“Why two stages — why not just retrieve the top 5 with the best model?” Strong answer: a bi-encoder must pre-compute document vectors, so query and document never interact; a cross-encoder models token-level interaction but costs one forward pass per candidate, which is unaffordable over a whole corpus. So: a shallow, fast, high-recall bi-encoder over $N\approx 100$, then an expensive cross-encoder over those $N$ to produce a precise $k\approx 5$–10. The reranker can only reorder what it is given, so $N$ sets its ceiling. Follow-up: “what if the reranker is your p99 bottleneck?” → distil it, use a late-interaction (ColBERT-style MaxSim) model, quantise it, or move to a smaller model and accept the precision loss — and measure it rather than assuming.

“When would you not use RAG?” Strong answer: three cases. (1) The corpus fits in the context window — under roughly 200k tokens, full context plus prompt caching is simpler and has no retrieval failure mode at all. (2) The task is reasoning or computation rather than lookup — “compute the growth rate”, “compare these two contracts clause by clause” — which needs tools or an agent loop, not a better retriever. (3) You are trying to change behaviour — format, tone, schema — which is fine-tuning, not retrieval. RAG changes what the model can see; fine-tuning changes what it does. Common wrong answer: “RAG is always better than fine-tuning because it is cheaper and needs no GPUs.” That conflates two orthogonal interventions.

“Your system is correct but users do not trust it. Why?” Strong answer: the chunk is the unit of citation, so a boundary that separates the answer from its context makes a correct answer uncitable. Also check that citations point at spans rather than whole documents; that the best passage is first rather than buried in the middle; and that abstention is a reachable state rather than a phrase the model has never been asked to emit. Follow-up: “how would you measure that?” → a human spot-check of citation precision — the fraction of cited spans that actually support the claim — measured separately from answer correctness.

“How do you handle a corpus that updates hourly?” Strong answer: decouple indexing from serving. Content-hash each chunk; on an update, re-embed only the changed chunks; store the embedding-model version in index metadata and refuse to serve a mismatch; use upsert semantics keyed on a stable chunk id. If the embedding model changes, it is a full re-index — budget for it, and deploy it blue/green rather than mutating in place. Common wrong answer: “just re-run the indexer every hour.” That re-embeds the whole corpus and creates a window where some chunks live in the new vector space and some in the old, so similarity scores are meaningless.

Next: Chunking — chunk boundaries set the recall ceiling before any model gets involved, which makes chunk size and overlap the highest-leverage knobs in the entire pipeline.