Retrieval: Hybrid Search and Reranking

TL;DR — Dense retrieval matches meaning; BM25 matches strings. They fail on disjoint queries: a bi-encoder blurs a rare token like ERR_5521 into subwords, while BM25 returns nothing when query and document share no vocabulary. Hybrid search runs both and fuses the lists with Reciprocal Rank Fusion, $\mathrm{RRF}(d)={\sum }_{r}1/(k+{\mathrm{rank}}_r(d))$ — ranks, not scores, so no cross-retriever normalisation is needed, and $k=60$ flattens the curve so consensus beats one retriever’s enthusiasm. A cross-encoder reranker then re-scores the top $N\approx 50$ candidates by running query and passage through one transformer together: the cheapest large quality win in the pipeline, and the most expensive thing you can add to p99. It cannot create a candidate — its ceiling is the first stage’s recall@$N$.

1. Why this exists

Two queries, two failures. Query A: “What changed for ERR_5521 in the 4.2 upgrade?” A vector index returns the semantically nearest chunks — troubleshooting pages about error codes in general, an upgrade guide, a logging page — and the chunk that literally contains ERR_5521 may not be in the top 50. The reason is mechanical: the token is rare, the tokenizer chops it into subwords (ERR, _55, 21), and the pooled embedding of a 300-word chunk is dominated by the other 299 words. Query B: “the fan keeps spinning up even when nothing is open.” BM25 returns nothing: the answering document says “thermal throttling under sustained idle load”, zero terms overlap, score zero.

These are not two bugs in two systems. They are the same structural property: lexical matching requires surface overlap, semantic matching requires that meaning survive the encoder’s bottleneck, and rare strings do not survive it.

Retrieval is really two jobs with different objectives: candidate generation — get the right chunk somewhere in a set of $N\approx 50\text{–}100$, measured by recall@$N$, served by a cheap model — and ordering — put it at the top of the $k\approx 5$ the generator reads, measured by nDCG@$k$ or MRR@$k$, served by a slow accurate one. Conflating them wastes months. If recall@100 is 0.62, reranking cannot get you past 0.62: it only permutes a list that mostly lacks the answer. If recall@100 is 0.95 and nDCG@5 is 0.51, a reranker is the highest return per line of code in the stack.

Where this matters for you. In an interview, “add hybrid search and a reranker” is the correct answer to a ranking problem and the wrong answer to a coverage problem. Say which one you measured before you say what you would add.

2. How it works

2.1 BM25, symbol by symbol

BM25 (Okapi BM25; the standard reference is Robertson & Zaragoza, The Probabilistic Relevance Framework: BM25 and Beyond) scores a document $d$ against a query $q$ as a sum over query terms:

$$BM25(q,d)=\sum _{t\in q}\mathrm{IDF}(t)\cdot \frac{f(t,d)\cdot (k_1+1)}{f(t,d)+k_1\left(1-b+b\frac{|d|}{\mathrm{avgdl}}\right)}$$

Every piece of that has a job:

• $t\in q$ — the summation runs over query terms only. BM25 is not symmetric: a document containing extra terms the query never asked for is not penalised. This is why BM25 is robust on long documents and why it cannot model negation, word order, or proximity.

• $f(t,d)$ — raw term frequency: how many times $t$ occurs in $d$. If you remove it, every document that contains the term at all scores identically and “which document discusses this most” is unanswerable. But raw $f$ alone is a bad signal, because the 20th occurrence of a word adds almost no evidence over the 19th. Hence the saturation below.

• $k_1$ (realistic range 1.2–2.0; Lucene and Elasticsearch default 1.2 — see Elasticsearch similarity settings) — the term-frequency saturation knob. As $f\to \infty$ the fraction tends to $k_1+1$, so with $k_1=1.2$ a term contributes at most $2.2\times$ its IDF however often it appears. The marginal value of each additional occurrence is

  $$g(f)-g(f-1)=\frac{k_1(k_1+1)}{(f-1+k_1)(f+k_1)},g(f)=\frac{f(k_1+1)}{f+k_1}$$

  for $b=0$. At $k_1=1.2$ the first occurrence is worth $1.0$, the second $0.375$, the third $0.196$: after three occurrences you hold $1.571$ of the asymptotic maximum $2.2$, i.e. 71% of the available credit. That is why $k_1$ exists — it is a derived saturating transform, not a fudge factor. Set $k_1=0$ and the fraction collapses to $f/f=1$: BM25 becomes binary term presence, and a document that mentions a term once outranks one that is about it. Set $k_1=100$ and saturation vanishes; keyword stuffing works again.

• $b$ (realistic range 0.5–0.8; default 0.75) — the document-length normalisation strength. The term $1-b+b|d|/\mathrm{avgdl}$ is the relative length penalty, call it $B$. At $|d|=\mathrm{avgdl}$, $B=1$ and the document is treated neutrally. At $|d|=2\mathrm{avgdl}$ with $b=0.75$, $B=0.25+1.5=1.75$, so the effective term frequency is divided by a larger number and the long document must mention the term proportionally more to win. Set $b=0$ and the length factor disappears: long documents win by sheer surface area, because they contain more of everything. Set $b=1$ and normalisation is total — a 20-token title and a 2000-token section are compared as if length were irrelevant, which over-penalises documents that legitimately discuss a topic at length. $b\approx 0.75$ is the empirical compromise between those two failure modes.

• $|d|$ and $\mathrm{avgdl}$ — document length in tokens and the mean over the collection, in whatever unit your analyzer produces. That is why $b$ is not portable across analyzers or languages: a tokenizer emitting CJK bigrams produces very different lengths from one emitting whitespace words, and the same $b$ will not mean the same thing.

• $\mathrm{IDF}(t)$ — inverse document frequency, the term’s informativeness, in the Lucene form (what Elasticsearch computes):

  $$\mathrm{IDF}(t)=\ln \left(1+\frac{N-n_t+0.5}{n_t+0.5}\right)$$

  $N$ is the collection size and $n_t$ the number of documents containing $t$. As $n_t\to N$ the IDF goes to ~0: a term in every document carries no information, so a match on it must not move the ranking. As $n_t\to 1$ with $N=10^6$, it is $\ln (1+666,667)\approx 13.4$ — a ~13-nat spread between the most and least informative terms, by far the largest dynamic range in the formula, and the reason stop-word removal is mostly redundant: stop words already have near-zero IDF. The $+1$ inside the log is deliberate. The textbook form $\ln \frac{N-n_t+0.5}{n_t+0.5}$ turns negative for $n_t>N/2$, which would make a frequent term punish a document for containing it; Lucene’s $+1$ floors it at 0.

What BM25 is not. The score is unbounded, grows with query length, and depends on collection statistics ($N$, $n_t$, avgdl) that change on re-index. So the same document can score 12 for one query and 3 for another (example), and its score moves when unrelated documents are added. That is exactly why fusing it with cosine similarity by score is fragile — see §2.3.

2.2 Why the dense half misses the string

A bi-encoder (DPR-style; Karpukhin et al. 2020) computes $E_q=f_{\theta }(q)$ and $E_d=f_{\theta }(d)$ independently and scores by cosine or dot product. Independence is what makes it fast — document vectors are precomputed offline, so query cost is one encoder pass plus an approximate nearest-neighbour lookup — and independence is also what makes it imprecise. Pooling destroys token identity: one vector summarises a whole chunk, so a rare identifier is a small perturbation among hundreds of tokens. Subword splitting destroys the string: ERR_5521 and ERR_5522 become nearly the same token sequence, so the exact digits the user cared about are what the representation is least sensitive to. And rare tokens get poor vectors, because they appear in few training pairs. That is not fixed by a bigger embedding model — domain fine-tuning with hard negatives is the real lever (see Embedding Models) — but a lexical index solves it exactly, for free, because it stores the string.

2.3 Fusion: ranks, not scores

You have two ranked lists and want one. The obvious move — normalise both score vectors and take a weighted average — fails three ways. The scales are not comparable (BM25 is unbounded and log-scaled; cosine lives in $[-1,1]$ with a useful range nearer $[0.3,0.9]$). Per-query normalisation forces every query’s best hit to 1.0 even if the best hit is bad, so a query where BM25 confidently separates the answer from 200 near-misses looks identical to one where it barely separates anything. And because cosine scores cluster tightly, max-normalising amplifies numerical noise into an apparent ordering.

Reciprocal Rank Fusion (Cormack, Clarke & Buettcher, SIGIR 2009) sidesteps all three by throwing the scores away and keeping only the order:

$$\mathrm{RRF}(d)=\sum _{r\in R}\frac{1}{k+{\mathrm{rank}}_r(d)}$$

• $R$ — the set of retrievers being fused (BM25, dense, and any others). A document absent from a retriever’s list contributes nothing from that retriever, which is the mechanism that rewards agreement.

• ${\mathrm{rank}}_r(d)$ — the 1-based position of $d$ in retriever $r$’s list. Ranks, not scores is the entire design. Rank is invariant to any monotone transform of a score, so BM25’s arbitrary scale and the embedding model’s compressed cosine range become irrelevant. That is why RRF needs no per-retriever tuning and why it is the default in Elasticsearch (RRF retriever docs: rank_constant defaults to 60) and in most vector databases. Note the 1-based indexing — using 0-based ranks divides by $k$ instead of $k+1$ at the top and makes the top-ranked item’s advantage slightly larger than intended.

• $k$ — the rank-smoothing constant. It is the only parameter, and its job is to decide how much consensus is worth versus enthusiasm. Compute the top-of-list contributions:

  $k$	rank 1	rank 2	rank 3	penalty for dropping one rank
  1	0.500	0.333	0.250	33%
  10	0.0909	0.0833	0.0769	8.3%
  60	0.01639	0.01613	0.01587	1.6%

  With $k=60$, being ranked 1st versus 3rd by one retriever is worth a 3% difference — nearly free. That is intentional: it means a document that both retrievers place at rank 3 scores $\frac{2}{63}=0.0317$, which beats a document that only one retriever places at rank 1 ($\frac{1}{61}=0.0164$) by roughly $2\times$. RRF is therefore a voting scheme: agreeing twice at moderate rank beats being loved once. Set $k=0$ (or $k$ very small) and enthusiasm dominates — the fused list is essentially the union of the two lists ordered by their individual top ranks, and you lose the cross-retriever signal you built the pipeline for. Set $k$ in the thousands and every rank contributes almost identically, so the fused order degenerates toward “documents retrieved by more retrievers first, ties arbitrary”. $k=60$ sits between those regimes and was the value used in the original paper; it is rarely worth tuning before you have a golden set (§7).

What RRF costs you. It discards magnitude: BM25 scoring 40 for an exact identifier match against a next-best 6 gets the same credit as a retriever barely separating noise. Where one retriever is known to be much stronger on a query class, weighted fusion can win — Elasticsearch supports per-retriever weight multipliers, giving ${\sum }_r{w}_r/(k+{\mathrm{rank}}_r(d))$. The evidence on RRF versus weighted score fusion is mixed and domain-dependent; see Know When to Fuse (2024) for a study where fusion choices did not transfer across languages. Measure $w_r$; do not believe it.

2.4 Reranking: give the query and the passage one model

A cross-encoder does not embed anything separately. It concatenates [CLS] query [SEP] passage [SEP], runs the pair through a transformer, and reads a scalar relevance score off the [CLS] head:

$$\mathrm{score}(q,d)=\sigma \left(w^{\top }{h}_{\text{[CLS]}}(q\oplus d)\right)$$

• $q\oplus d$ — the concatenation. This is the crucial difference from a bi-encoder: both texts are in the same forward pass, so every query token can attend to every passage token across all layers. The model can implement “the number after ERR_ must match”, “the answer must be in the same sentence as the negation”, and so on. A bi-encoder structurally cannot: it commits to a document vector before it has seen the query.
• $h_{\text{[CLS]}}$ — the pooled representation the classification head reads. Because every layer’s self-attention lets this position aggregate information from the whole pair, it is a joint query-document summary.
• $w,\sigma$ — a learned linear head and the logistic function. Removing $\sigma$ changes nothing about the ranking (it is monotone), which is why many rerankers ship raw logits; it matters only if you want to threshold on “is this relevant at all”, because then you need calibrated probabilities, and raw logits are not calibrated across queries.

The cost is the point: nothing can be precomputed on the document side. Every candidate needs its own forward pass at query time, so reranking $N$ candidates costs $N$ transformer forwards of length $|q|+|d|$ — $O(N)$ query-time work where the bi-encoder was $O(1)$.

Common rerankers [as of 2026-10]: BAAI/bge-reranker-v2-m3 (XLM-RoBERTa-large class, multilingual, usable via sentence-transformers’s CrossEncoder), BAAI/bge-reranker-large, mixedbread-ai/mxbai-rerank-large-v1, and the hosted Cohere Rerank endpoint — Rerank 3.5 has a 4096-token context (Cohere release notes, Dec 2024). Model names move; pin the checkpoint you measured.

Late interaction (ColBERT) sits between the two on the cost/accuracy curve: Khattab & Zaharia 2020 keeps one vector per token and scores with MaxSim, $\mathrm{score}(q,d)={\sum }_{i\in q}{\max }_{j\in d}(E_{q_i}\cdot E_{d_j})$, where $E_{q_i}$ and $E_{d_j}$ are contextual token embeddings. The ${\max }_j$ lets each query token pick the document token it matches best instead of being averaged into one pooled vector; the ${\sum }_i$ aggregates over the query. Document vectors are precomputed, so query cost is far below a cross-encoder’s — the price is one vector per document token, roughly $10^2\times$ the dense index.

2.5 The whole pipeline

flowchart LR
    Q["query"] --> BM["BM25 / Lucene index"]
    Q --> EM["bi-encoder"]
    EM --> ANN["ANN search over doc vectors"]
    BM --> L1["ranked list (lexical)"]
    ANN --> L2["ranked list (dense)"]
    L1 --> RRF["RRF fusion, k=60"]
    L2 --> RRF
    RRF --> N["top N ≈ 50 candidates"]
    N --> CE["cross-encoder rerank"]
    CE --> K["top k ≈ 5"]
    K --> CTX["context assembly"]

Two properties of that diagram matter. The reranker sits after fusion, on the union, not on each list separately. And the two retrievers are independent, so they can be issued concurrently — fusion waits for the slower one, so hybrid p50 is $\max$ of the two retrieval latencies, not their sum.

3. Implementation

A real, runnable hybrid retriever. The lexical half is in-process so the code is self-contained; in production you would issue it against Lucene/Elasticsearch/OpenSearch or a Postgres tsvector index and inherit its analyzer, posting-list optimisations, and incremental updates.

"""
Hybrid retrieval: BM25 + dense bi-encoder, fused with RRF, then cross-encoder
rerank. Pinned dependencies [as of 2026-10]:
    pip install "rank_bm25==0.2.2" "sentence-transformers==5.4.0" "numpy<3"

rank_bm25 is a pure-Python teaching implementation: fine for tens of thousands
of chunks, wrong for production. Swap in a real IR engine before benchmarking.
"""
from __future__ import annotations

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder, SentenceTransformer

EMBED_MODEL = "BAAI/bge-base-en-v1.5"      # 768-d bi-encoder
RERANK_MODEL = "BAAI/bge-reranker-v2-m3"   # cross-encoder


def tokenize(text: str) -> list[str]:
    """Placeholder analyzer: lowercase + whitespace split.

    Index-time and query-time tokenization must use the *same* analyzer
    (Lucene's StandardAnalyzer lowercases and splits on punctuation/digits).
    A mismatch drops recall silently and raises nothing.
    """
    return text.lower().split()


def rrf_fuse(rankings: list[list[int]], k: int = 60) -> list[tuple[int, float]]:
    """Reciprocal Rank Fusion over best-first lists of document indices.

    A document missing from a list contributes nothing from that list.
    """
    fused: dict[int, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):   # 1-based!
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda kv: -kv[1])


def top_n(vec: np.ndarray, n: int) -> list[int]:
    """Indices of the n largest entries, best-first. O(N), not O(N log N)."""
    if n >= len(vec):
        return list(np.argsort(-vec))
    idx = np.argpartition(-vec, n)[:n]
    return list(idx[np.argsort(-vec[idx])])


class HybridRetriever:
    def __init__(self, chunks: list[str]):
        self.chunks = chunks
        self.bm25 = BM25Okapi([tokenize(c) for c in chunks])       # offline
        self.embedder = SentenceTransformer(EMBED_MODEL)
        self.doc_vecs = self.embedder.encode(
            chunks,
            normalize_embeddings=True,          # makes dot product == cosine
            convert_to_numpy=True,
            batch_size=64,
        )
        self._reranker: CrossEncoder | None = None   # lazy: GB-scale checkpoint

    def retrieve(self, query: str, n_candidates: int = 50) -> list[int]:
        lex = top_n(np.asarray(self.bm25.get_scores(tokenize(query))), n_candidates)
        q = self.embedder.encode([query], normalize_embeddings=True)[0]
        den = top_n(self.doc_vecs @ q, n_candidates)
        return [d for d, _ in rrf_fuse([lex, den], k=60)][:n_candidates]

    def search(self, query: str, n_candidates: int = 50, k: int = 5):
        cand = self.retrieve(query, n_candidates)
        if self._reranker is None:
            self._reranker = CrossEncoder(RERANK_MODEL)
        scores = np.asarray(
            self._reranker.predict(
                [(query, self.chunks[i]) for i in cand], batch_size=16
            )
        )
        order = np.argsort(-scores)[:k]
        return [(cand[i], float(scores[i])) for i in order]

The knobs that matter, and what moving each one does:

Two notes on the code. np.argpartition is used instead of a full sort because only the top $n$ matter and it is linear rather than $n\log n$ — irrelevant at 10k chunks, meaningful at 50M. And normalize_embeddings=True is what makes the dot product equal cosine similarity; mixing a normalised index with an unnormalised query is a classic silent bug.

4. Numbers and trade-offs

The reranker is where the FLOPs go

A transformer forward pass costs roughly $2PT$ FLOPs for $P$ parameters and $T$ tokens (2 = one multiply and one add per parameter per token; at $T\approx 512$ the attention $T^2$ term is small next to the MLP term for a 24-layer encoder). For a ~$5.7\times 10^8$-parameter cross-encoder (bge-reranker-v2-m3; [UNVERIFIED: exact parameter count]) scoring 512-token passages:

$$2\times 5.7\times 10^8\times 512\approx 5.8\times 10^{11}\text{FLOP per candidate}$$

At $N=50$ that is $\approx 2.9\times 10^{13}$ FLOPs — 29 TFLOPs per query. The dense stage, by contrast, is one query encode ($2\times 1.1\times 10^8\times 16\approx 3.5\times 10^9$ FLOP) plus an exact dot product over $10^6$ documents ($\approx 1.5\times 10^9$ FLOP) — about $5\times 10^9$ FLOPs. Reranking 50 candidates costs roughly 6000× the compute of the retrieval that produced them, all on the critical path.

Converting to wall-clock needs an assumed effective throughput. At ~25% of an A100’s 312 TFLOP/s fp16 peak — realistic for small-batch encoder inference — 29 TFLOPs is ~0.37 s (example; measure on your hardware and passage lengths). Published reranker latencies for 50–100 candidates land in the low hundreds of milliseconds on a GPU, worse on CPU, and hosted APIs add a round trip.

Index and latency budgets

A dense index of $N=10^6$ chunks at $d=768$ costs $Nd\times 4\approx 3.07$ GB in fp32, 1.54 GB in fp16, 0.77 GB in int8, plus ~0.13 GB of HNSW links at $M=32$. An illustrative end-to-end breakdown for that corpus with a local GPU (example — replace with your own measurements):

The conclusion is not “skip the reranker” — it is “choose $N$ where recall@$N$ saturates, not where the latency budget dies”. Plot recall@$N$ for $N\in \{10,20,50,100,200\}$; it almost always has a knee, and reranking past the knee costs linearly and buys nothing.

5. Pitfalls

Normalising scores before fusing, or fusing with 0-based ranks. Min-max or z-score normalisation per query, then averaging: symptom is a hybrid no better than the better single retriever, with weights that need retuning as the query mix moves. Zero-based ranks or k=0: symptom is a fused list where whichever retriever had the best top hit wins, consensus documents never surface, and k=0 raises ZeroDivisionError on the first document — the off-by-one in enumerate(..., start=1) is the most common RRF bug there is.

Reranking each list before fusion. Symptom: identical top-k to fusion alone but $2N$ cross-encoder forwards — double latency for nothing, because RRF discards the reranker’s scores anyway. Rerank the union, after fusion.

Truncating passages at the reranker’s default max length. Many CrossEncoder wrappers truncate at the checkpoint’s max_length, which for some models is well below what the checkpoint supports. Symptom: long-document queries plateau — adding candidates stops helping, and failures cluster on the longest chunks. Print the tokenized candidate length distribution and compare it to model.max_length before blaming the model.

Assuming the vector index is exact. HNSW and IVF are approximate; their recall is a parameter (ef_search, nprobe). Symptom: offline recall@$k$ looks fine because the eval harness uses the same approximate index, while users report the answer “exists but was not found”; or recall jumps when you turn a dial and change no code. Fix: brute-force exact search on a query sample to get a ground truth, then measure ANN recall against it.

Thresholding on reranker scores as if they were probabilities. Symptom: the “insufficient evidence → abstain” path fires on 4% of queries with reranker A and 19% with reranker B, with no corpus change. Logits are not calibrated across models or queries; fit a threshold per model on held-out data, or use a separate relevance classifier.

Not monitoring the two retrievers separately. Symptom: a corpus change silently breaks one retriever and hybrid quality drops a few points — too small to notice in the fused metric. Log lexical-only and dense-only recall.

Tuning the generator when retrieval is the bottleneck. Symptom: a week of prompt wording moves answer accuracy ±2 points, nobody has measured recall@$k$, and the old prompt answers correctly given the gold chunk. Run the gold-context test first — see RAG Evaluation §2.4.

6. Using it in a project

Add the lexical half when queries contain identifiers, error codes, names, acronyms, product SKUs, version numbers or quoted phrases; when the corpus is large and heterogeneous; or when you cannot fine-tune the embedder and need free recall. The marginal cost is one more index.

Add a reranker when the right chunk is usually in the top $N$ (recall@$N$ high) but not at the top (nDCG@5 or precision@5 low). That is the signature of an ordering problem, and a cross-encoder is the best cost-per-point fix available.

Do not add a reranker when any of these hold:

• The latency budget is tight and there is no GPU. A voice or autocomplete surface with a 300 ms end-to-end budget cannot absorb 200–400 ms. Use a late-interaction model, a distilled 6–12-layer cross-encoder, or spend the tokens on more context instead.
• Recall@$N$ is the bottleneck. At recall@100 of 0.62 the reranker is rearranging a list that mostly lacks the answer. Fix chunking (Chunking), the embedder, or query transformation first.
• The query distribution has no lexical signal. If every query is a natural paraphrase and no literal string ever matters, BM25 contributes near-random candidates that dilute the fused list. Check lexical-only recall on the golden set: if it is at chance, drop it.
• The corpus is small. Under roughly $10^4$ chunks, a strong bi-encoder with a reranker over everything often beats a hand-tuned hybrid, with one fewer index to operate.
• The first stage is already strong. Reranking gains most where the first stage is weak; if you already fine-tuned with hard negatives, measure before assuming it still pays.

The honest default. Measure the pipeline you have. Many systems that “disappoint at generation” have a chunk-boundary defect at retrieval time.

7. How you would prove it worked

Define the metrics and the ceiling. Recall@$N$: with relevant-chunk set $G$ and top-$N$ list $R_N$, $\mathrm{recall}@N=|G\cap R_N|/|G|$ — did the answer-bearing chunk make the candidate list? This is the ceiling on everything downstream. nDCG@$k$: is the ordering good, with graded penalties for burying the answer (definition in RAG Evaluation §2.1)? MRR@$k$: how high is the first relevant chunk — the right headline when one chunk usually suffices. End-to-end answer correctness: the only metric a user feels, and the one that can move for reasons unrelated to retrieval.

Run a staged ablation, not a single A/B. Each row adds one component, so you learn where the recall comes from:

Two things this table must show: recall stops moving at the reranker row (that is what reranking is, and reporting it prevents over-claiming), and the latency column makes the trade explicit. Break results down by query class — (a) identifier or quoted-string queries, (b) conversational/paraphrase, (c) multi-hop. Hybrid should help (a) and be roughly neutral on (b); if it hurts (b), BM25 is adding noise or the analyzer is wrong. Then guard against overfitting: k1, b, RRF k, and $N$ are four parameters tuned against a few hundred questions. Tune on a dev split, report on a held-out split.

8. Interview drill

“Why is BM25 still around when we have embedding models?” Strong answer: BM25 is an exact string matcher with a principled term-weighting scheme; embeddings are lossy summaries that systematically fail on rare tokens, identifiers and subword-split strings. They are complementary and fail on disjoint query sets, which is why hybrid wins on BEIR-style benchmarks. Common wrong answer: “BM25 is faster / cheaper.” True, but it answers a different question — you would use it even if it were slower, because it retrieves what the dense model cannot.

“Why RRF rather than averaging normalised scores?” Strong answer: BM25 scores and cosine similarities are not on the same scale, and per-query normalisation makes the mapping depend on the query’s own score spread, so a confident query and a hopeless one look identical afterwards. Ranks are invariant to monotone transforms, so RRF needs no calibration. Common wrong answer: “RRF is more accurate.” It is more robust, not more accurate in general — it discards magnitude, and where one retriever is much stronger, weighted score fusion can beat it. Follow-up — “why 60?”: $k$ sets the exchange rate between consensus and top-rank evidence. At $k=60$ the gap between rank 1 and rank 3 is about 3%, so a document that both retrievers place at rank 3 outscores one that only a single retriever puts first. It is the original paper’s value and Elasticsearch’s default; it is the right order of magnitude, and tuning it before you own a golden set is premature. The wrong answer is “60 is the optimal constant” — the paper used 60, which is a different claim.

“How many candidates should you rerank?” Strong answer: plot recall@$N$ against $N$ and find the knee; rerank to the knee, because latency is linear in $N$ and reranking past it buys no achievable quality. That also bounds the reranker’s maximum possible improvement: it can only permute the candidate set it was given, so if recall@50 is 0.8, no reranker produces an answer-bearing chunk for the other 20% of queries. Follow-up: “what if recall@$N$ never saturates?” → the first stage is the problem; invest in chunking, the embedder, or query transformation instead.

“Cross-encoder vs bi-encoder — what is the architectural difference, and why does it cost more?” Strong answer: the bi-encoder encodes query and document independently, so document vectors are precomputed; the cross-encoder runs the concatenated pair through one transformer, so every query token attends to every document token, at the cost of one forward pass per candidate at query time — $O(N)$, not $O(1)$. Common wrong answer: “the cross-encoder is bigger.” Size is incidental; the interaction pattern is the difference.

“Your hybrid search made things worse. What happened?” Strong answer: in order of likelihood — index-time and query-time analyzers do not match, so the lexical half returns noise; the fusion is score-based with per-query normalisation; or BM25 is uninformative on this query distribution and dilutes a fusion window that is too small. Diagnose by evaluating lexical-only and dense-only recall separately on the same golden set.

Next: Query Transformation — when the query and the document share neither vocabulary nor meaning, no amount of fusion helps, and the fix moves to the query.