Query Transformation

TL;DR — Retrieval fails when the query and the document do not look alike, and much of the time that is the query’s fault, not the index’s: it is a pronoun (“what about its salary?”), a paraphrase with none of the document’s vocabulary, or a multi-hop question no single chunk answers. Query transformation fixes retrieval by rewriting the query before it reaches the index. The four moves are rewriting (make a conversational query self-contained), HyDE (generate a hypothetical answer, embed that, and retrieve in answer space rather than question space), multi-query expansion (several paraphrases, fused with RRF), and decomposition (split a multi-hop question into sub-questions). Every one of them costs at least one extra LLM call, so the discipline is to route: transform only the queries that need it, and only after recall@$k$ has proven that retrieval is the bottleneck.

1. Why this exists

Three queries, three different reasons the index comes back empty.

The elided query. A user asks “how much does the Pro plan cost?”, gets a good answer, then asks “and what about the annual one?” A single-shot dense retriever embeds a string about annualness and returns renewal policies, tax-year documents, and an annuity product page. The words “Pro plan” and “price” are gone, and nothing in the index can recover them. This is not a retrieval-quality problem; it is a query-construction problem.

The vocabulary mismatch. The user writes “my laptop gets really hot and the fan is always on”; the answering passage says “thermal throttling occurs when sustained load exceeds the cooling envelope”. A lexical retriever scores zero. A dense retriever usually copes — that is what embeddings are for — but in a domain corpus (clinical notes, contracts, internal jargon) a general-purpose embedder may not have learned the mapping either. Reformulating the query in the corpus’s own vocabulary is cheaper than fine-tuning an embedder.

The multi-hop question. “Which of the two vendors we signed in 2023 had a SOC 2 report that expired last year?” needs three facts — the vendor list, the 2023 signings, the expiry dates — from three documents. No single chunk answers it, so no retriever configuration can return an answering chunk. Asking the retriever harder cannot help; the question has to be taken apart.

There is also an organisational reason: query transformation is the cheapest place to intervene. Fixing chunking means re-indexing; fixing an embedder means labelling data and retraining; a rewrite step is one prompt and a latency budget.

But. Every transformation here is an extra LLM call on the critical path, and every one can lower recall: a rewriter that hallucinates a brand name, a HyDE document about the wrong product, a decomposition of a question that was single-hop all along. The default is off, turned on per query class after measurement.

2. How it works

2.1 Rewriting: make the query self-contained

The job is coreference and ellipsis resolution against the conversation, not “improving” the query. Formally, given the conversation history $H=(u_1,a_1,\dots ,u_{t-1},a_{t-1})$ and the current turn $u_t$, produce $q^{\prime }=g(H,u_t)$ such that $q^{\prime }$ is interpretable without $H$:

$$q^{\prime }=\arg \underset{q}{\max }{\mathbb{E}}_{d\sim {\mathcal{R}}_q}\left[\mathrm{rel}(d,u_t\mid H)\right]$$

• $g$ — the rewriter, an instruction-following LLM constrained by a prompt that permits only substitution, never addition of facts. That constraint is the whole design: an unconstrained “rewrite this to be a better search query” prompt invents requirements, and an invented requirement becomes an unfalsifiable filter on the index.
• $H$ — the conversation history, the only source of the missing referents. If the antecedent is not in $H$, the correct behaviour is to ask a clarifying question rather than guess — see Intent Recognition and Routing for how that decision is usually routed.
• ${\mathcal{R}}_q$ — the result set the retriever returns for candidate query $q$. The expectation is over documents, and $\mathrm{rel}(d,u_t\mid H)$ is the original user intent’s relevance, not the rewritten string’s relevance. You are optimising $q^{\prime }$ against a target you cannot observe, which is why the only honest evaluation is end-to-end recall on a golden set that preserves the conversational context.

This belongs in the retrieval path, not the generator. If you pass the raw turn to the retriever and let the generator paper over it, the retriever returns unrelated chunks, the generator ignores or hallucinates around them, and a cheap retrieval bug has become an expensive generation bug that is much harder to diagnose.

2.2 HyDE: retrieve in answer space

HyDE (Gao, Ma, Lin & Callan 2022) observes that queries and documents live in different regions of embedding space even when they are relevant to each other. A question is short, interrogative and abstract; a passage is long, declarative and specific. A bi-encoder trained on query–passage pairs learns some mapping between the regions, but with no relevance labels the mapping is thin.

So generate the missing half:

$$\hat{d}=\arg \underset{d\in \mathcal{C}}{\max }⟨E(g(q)),E(d)⟩$$

• $q$ — the (already rewritten) user query.
• $g(q)$ — a hypothetical document produced by an instruction-following LLM: a fluent passage that would answer $q$ if it were true. It is explicitly not required to be factually correct — the paper’s framing is that the document “captures relevance patterns but is unreal and may contain false details”.
• $E$ — the same document encoder used for the corpus, applied to $g(q)$. Critically, the same one: HyDE is a change of query representation, not a change of index. The query is no longer encoded by the query-side head at all.
• $\mathcal{C}$ — the corpus. Search is the ordinary nearest-neighbour search you already had.
• $⟨\cdot ,\cdot ⟩$ — the similarity the index uses (cosine on normalised vectors, or the inner product the ANN supports).

The mechanism that makes this work is the encoder’s information bottleneck. $g(q)$ is a long, fluent, partly fabricated passage; its embedding is a pooled summary, so fabricated specifics — a wrong part number, an invented date — are averaged away while the topical structure survives and lands the vector near real passages about that topic. That is why a wrong hypothetical document can still retrieve a right real one. Remove the bottleneck (retrieve with token-level late interaction against the raw hypothesis) and the fabricated details start to hurt.

HyDE’s reported result is a large zero-shot gain over Contriever, an unsupervised dense retriever, with performance “comparable to fine-tuned retrievers” across web search, QA and fact verification, in several languages. Two costs are immediate: one LLM call per query, and the query-side encoder becomes unusable — you cannot precompute anything for a vector you have not generated yet, so the LLM call sits on the latency path.

When HyDE helps and when it does not. It helps on short, abstract questions against a prose corpus with no labelled data. It hurts when the query is already answer-shaped — a quoted identifier, a product code, a question whose answer is a single string — because hallucinating a passage around ERR_5521 adds no information the token did not carry, and the fiction can drift the vector away from the exact-match document. It also hurts when the generator’s domain knowledge is poor: a model that does not know the domain writes fluently about the wrong topic, and the bottleneck rescales the fiction rather than filtering it.

2.3 Multi-query expansion and fusion

Generate $m$ paraphrases of the query and retrieve for each; the union is the candidate set. The objective is a recall objective, and it is worth writing down because it tells you how many paraphrases are worth buying. Let $p$ be the per-paraphrase recall — the probability that a single paraphrase retrieves the answer-bearing chunk somewhere in its top $k$ (miss, not error: a paraphrase can still return it at rank 40). If the paraphrases were independent, the union’s miss probability would be $(1-p{)}^m$, so

$${\mathrm{recall}}_m=1-(1-p{)}^m$$

The independence assumption is false, and that is the point of the table. Paraphrases from the same LLM on the same query correlate: if a paraphrase misses because the concept is absent from the corpus, every paraphrase misses. The realistic model is a mixture — an independent chance of catching a different facet, plus a shared failure mode. So measure $p$ and the conditional miss rate $\mathrm{Pr}(\text{all }m\text{ miss})/(1-p)$ on a golden set. If that conditional rate stays near $1$ — paraphrases miss exactly when the original would — the variants are too similar to be worth the retrievals: diversify the prompt toward different aspects and vocabulary, or stop.

Fusion reuses Reciprocal Rank Fusion — the same function, with $m$ lists instead of two:

$$\mathrm{RRF}(d)=\sum _{r=1}^m\frac{1}{k+{\mathrm{rank}}_r(d)}$$

• $m$ — the number of generated variants. Cost is linear: $m$ retrievals plus one LLM call for all of them. $m\in \{3,5\}$ is the usual range; the paper-style name for this pattern is RAG-Fusion (a community term, not a paper result).
• $k$ — the same smoothing constant, same justification: with $k=60$, a document retrieved by several paraphrases at moderate rank beats a document that only one paraphrase loves, which is exactly the signal you are buying. RRF needs no score calibration, which matters here because all $m$ lists come from the same retriever and do have comparable scores — a naive union by score would collapse to whichever paraphrase produced the largest-magnitude similarity, and similarity magnitudes are not comparable across queries.
• ${\mathrm{rank}}_r(d)$ — the 1-based rank in paraphrase $r$’s list. A document absent from a list contributes nothing from it.

Note what this does not do: it does not change the ranking model, so it cannot fix a paraphrase-blind embedder. It buys coverage.

2.4 Decomposition: one question, several retrievals

Multi-hop questions fail structurally. If answering requires the composition $f(a,b)$ where $a$ and $b$ live in different chunks, then a single retrieval call has to return both, and the second one is only findable once you know the first. The iterative form (IRCoT, Trivedi et al., ACL 2023) makes that dependency explicit:

• Retrieve with the original question, read the top passages.
• Generate one reasoning step (a sentence of chain-of-thought) grounded in them.
• Use that sentence as the next retrieval query, because “what to retrieve depends on what has already been derived”.
• Repeat until the answer is produced or a step budget is exhausted.

The paper reports up to 21 points better retrieval and up to 15 points better downstream QA across HotpotQA, 2WikiMultihopQA, MuSiQue and IIRC, with GPT-3 and no training. Those numbers come from a specific setup (multi-hop QA benchmarks with gold supporting facts), so read them as evidence that the mechanism is real, not as a forecast for your corpus.

The simpler, non-iterative variant is decompose-then-retrieve: ask the LLM to split the question into sub-questions up front, retrieve and answer each, then synthesise. One extra call and no loop, which matters for a bounded p99. It fails when the sub-questions are not independent — the classic failure is discovering that the second sub-question was the wrong one to ask given the first answer.

2.5 Which one to use

flowchart TD
    Q["incoming query"] --> CONV{"depends on earlier turns?"}
    CONV -->|yes| RW["rewrite for coreference"]
    CONV -->|no| HOP
    RW --> HOP{"needs facts from >1 document?"}
    HOP -->|yes| DEC["decompose / iterate (IRCoT)"]
    HOP -->|no| VOC
    DEC --> VOC{"does it share vocabulary with the corpus?"}
    VOC -->|no| HYDE["HyDE: embed a hypothetical answer"]
    VOC -->|yes| RECALL{"recall@k already adequate?"}
    HYDE --> RECALL
    RECALL -->|yes| RET["retrieve as-is"]
    RECALL -->|no| MQ["multi-query expansion + RRF"]
    MQ --> RET

Read the diagram as a filter, not a pipeline. Most queries should exit at the bottom with no transformation at all. Each gate is a question you can answer from cheap signals: depends on earlier turns is “is the history non-empty and does the turn contain a pronoun or an elided noun phrase”; needs >1 document can be approximated by a small classifier or by whether the query contains a comparison or a causal connectives pair; shares vocabulary is a lexical-overlap score. The first gate is nearly free and always worth it. The last two are worth it only after measurement.

3. Implementation

The functions below take an injected llm callable, so the code is testable without a network and portable across providers. The concrete binding uses the openai Python client (openai>=1.0; verify the installed major version [as of 2026-10]) and sentence-transformers==5.4.0 for encoding.

"""
Query transformation: rewrite, HyDE, multi-query expansion.

    pip install "openai>=1.0" "sentence-transformers==5.4.0" "numpy<3"
"""
from __future__ import annotations

import os
from typing import Callable

import numpy as np
from openai import OpenAI
from sentence_transformers import SentenceTransformer

LLM = Callable[[str], str]           # prompt -> completion text


def openai_llm(model: str = "gpt-4o-mini", temperature: float = 0.0) -> LLM:
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

    def call(prompt: str) -> str:
        r = client.chat.completions.create(
            model=model,
            temperature=temperature,
            messages=[{"role": "user", "content": prompt}],
        )
        return r.choices[0].message.content.strip()

    return call


# ---------------------------------------------------------------- 1. rewrite
REWRITE_PROMPT = """\
Rewrite the user's latest message into a standalone search query.

Rules:
- Resolve pronouns and elided noun phrases from the conversation.
- Use ONLY information present in the conversation. Do not add facts,
  constraints, brands, dates or numbers that the user did not state.
- If a referent cannot be resolved, output exactly: CLARIFY
- Output the query only. No preamble, no quotes.

Conversation:
{history}

Latest message: {turn}
Standalone query:"""


def rewrite(llm: LLM, history: list[str], turn: str) -> str:
    out = llm(REWRITE_PROMPT.format(history="\n".join(history), turn=turn))
    if out.upper().startswith("CLARIFY"):
        raise ValueError("unresolvable reference: ask the user")
    return out


# -------------------------------------------------------------------- 2. HyDE
HYDE_PROMPT = """\
Write a short passage that would answer the question if it were true.
Be concrete and use the vocabulary a document on this topic would use.
Do not hedge, do not explain, do not mention that this is hypothetical.
Answer in 3-5 sentences.

Question: {query}
Passage:"""


def hyde_embed(embedder: SentenceTransformer, llm: LLM, query: str) -> np.ndarray:
    """Embed a hypothetical answer instead of the question."""
    doc = llm(HYDE_PROMPT.format(query=query))
    return embedder.encode([doc], normalize_embeddings=True)[0]


# ----------------------------------------------------- 3. multi-query + RRF
MULTIQUERY_PROMPT = """\
Generate {m} search queries that are diverse in wording but all aim to find
the same information as the user's question. Vary the vocabulary; do not just
reorder words. One per line, no numbering.

Question: {query}"""


def multi_query(llm: LLM, query: str, m: int = 4) -> list[str]:
    out = llm(MULTIQUERY_PROMPT.format(m=m, query=query))
    variants = [ln.strip(" -0123456789.") for ln in out.splitlines() if ln.strip()]
    return [query] + variants[:m]      # always keep the original


def rrf_fuse(rankings: list[list[int]], k: int = 60) -> list[tuple[int, float]]:
    """Same function as the hybrid chapter: 1-based ranks, ranks not scores."""
    fused: dict[int, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            fused[doc_id] = fused.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(fused.items(), key=lambda kv: -kv[1])


def multi_query_retrieve(retrieve, llm: LLM, query: str,
                         m: int = 4, n: int = 50) -> list[int]:
    """`retrieve(q, n) -> list[int]` is your existing first-stage retriever."""
    rankings = [retrieve(v, n) for v in multi_query(llm, query, m)]
    return [doc_id for doc_id, _ in rrf_fuse(rankings, k=60)][:n]


# ------------------------------------------------------------ 4. decomposition
DECOMPOSE_PROMPT = """\
Break the question into the smallest set of independent sub-questions whose
answers together determine the answer. At most {max_steps}. If the question is
already single-hop, return it unchanged. One per line, no numbering.

Question: {query}"""


def decompose(llm: LLM, query: str, max_steps: int = 4) -> list[str]:
    out = llm(DECOMPOSE_PROMPT.format(max_steps=max_steps, query=query))
    subs = [ln.strip(" -0123456789.") for ln in out.splitlines() if ln.strip()]
    return subs[:max_steps] if subs else [query]

The knobs, and what moving each one does:

Two details in the code are deliberate. The rewrite prompt’s failure mode is an explicit token (CLARIFY) rather than a silent guess — the caller must decide between asking the user and falling back to the raw turn, and that decision should not be made by the model. And multi_query always keeps the original query in the list, because a set of paraphrases that all drift is strictly worse than the query the user actually typed.

4. Numbers and trade-offs

Latency and cost are additive and land on p99. A transformation step is a generation, and generations are slow because they are autoregressive. Rough budgets for one call producing ~60 output tokens (example):

Compare with the retrieval you are trying to improve: 40–60 ms. A rewrite step can easily cost 10× the retrieval it feeds, which is why §6 exists. Token cost is the smaller problem — a rewrite is roughly $T_{\text{prompt}}+T_{\text{out}}$ tokens, maybe 150 + 30, cheap at mid-tier prices. Latency and the extra failure mode are the real bill.

The quality numbers you can actually cite. HyDE reports a large zero-shot recall gain over Contriever and parity with fine-tuned retrievers on web search, QA and fact verification (Gao et al. 2022). IRCoT reports up to +21 retrieval and +15 QA points on multi-hop benchmarks (Trivedi et al. 2023). Both were measured on retrieval benchmarks with relevance labels, not on your corpus. A recent applied comparison of query-fusion and multi-query variants in RAG systems exists (ACM, 2025) but its findings are not verified here [UNVERIFIED: results not read].

The trade, in one line each: rewrite buys correctness on conversational queries at the cost of one call and hallucinated constraints; HyDE buys zero-shot recall on abstract queries and forfeits the precomputed query path; multi-query buys coverage at $m$ retrievals; decomposition buys multi-hop answerability at the cost of sequential calls.

5. Pitfalls

An unconstrained rewriter that adds constraints. The prompt says “rewrite to improve search” and the model adds “2024” or a product tier the user never mentioned. Symptom: recall drops on conversational queries relative to using the raw turn, and the failing queries all contain a keyword absent from the user’s own words. Fix: constrain to substitution-only, temperature 0, plus a guard — if the rewrite introduces content words absent from both the history and the turn, fall back to the raw turn.

Transforming a query that should not be transformed. A user pastes a stack trace and the rewriter summarises it into “error with Java application”; a user searches ERR_5521 and HyDE writes a paragraph about error handling in general, pulling the vector away from the exact-match document. Symptom: with the transformation on, exact-match queries get worse while abstract queries get better, so the aggregate metric barely moves and the regression is visible only per query class. Fix: route by query type — never rewrite or HyDE a query that is already a literal string lookup.

Multi-query variants that are the same query, fused by score. The LLM emits five reorderings, and because all variants come from one retriever, someone takes the max score as the union. Symptom: recall after fusion is nearly identical to the original alone while latency is $m\times$, and results are unstable across rephrasings. Fix: measure the conditional miss rate (§2.3), diversify the prompt toward different aspects rather than synonyms, and fuse with RRF.

Decomposing single-hop questions. The model dutifully emits three sub-questions for “what is the refund window?” Symptom: latency triples and accuracy is unchanged; sub-question 2’s answer contradicts sub-question 1 because they were never independent. Fix: gate decomposition on a multi-hop signal, cap max_steps, and always keep the original question in the retrieval list.

Letting the IRCoT loop run unbounded. Each step feeds the next retrieval. Symptom: a small fraction of queries take 10–20 seconds and dominate p99, and occasionally the chain drifts to an unrelated topic and never returns. Fix: a hard step budget plus a similarity check between the current step and the original question to detect drift.

6. Using it in a project

When to reach for this. Three signals justify it, in increasing strength: (1) the query is conversational — if the retriever sees only the last turn of a multi-turn chat, rewriting is usually an unconditional win and the first thing to build; (2) recall is the bottleneck and it is a vocabulary problem — you have measured recall@$k$, and the misses are queries where a human can see the answer in a chunk but no query term or embedding neighbour reaches it; that is exactly what HyDE and multi-query address; (3) the question is demonstrably multi-hop — comparisons, “which of the two”, aggregations over entities, causal chains. Decomposition is the only technique here that changes the answerability of the question, because it changes what is retrieved rather than how well.

When NOT to.

• Latency budget under ~500 ms end to end. A rewrite alone can consume that. Use a fine-tuned small rewriter (a 1–3B model) or a distilled classifier, or handle coreference with a cheap rule: if the turn contains a pronoun, prepend the previous turn’s nouns.
• Recall is already adequate. At recall@10 of 0.9 you have a ranking problem; spend the latency on a reranker (Retrieval: Hybrid Search and Reranking), not on generating more candidates.
• Queries are uniform, well-formed or literal. A keyword search box, an API where the caller constructs the query, code search, log search, SKU lookup — no ellipsis to resolve, and rewriting is pure risk.
• You have no golden set. You cannot tell whether a transformation helped, and each one can hurt. Build the eval first — see RAG Evaluation.
• The corpus is tiny. With 5k tokens of documents there is nothing to retrieve; put them all in the prompt.

The routing pattern that works. Gate transformations by query class (conversational vs literal, abstract vs identifier, single-hop vs multi-hop) and apply the cheapest sufficient one. A router that sends 30% of traffic through a rewrite beats a pipeline that runs three LLM calls on every query.

7. How you would prove it worked

Baseline first, and make it the raw user turn — not the rewritten turn, not the previous system’s output. The comparison is retrieve(u_t) versus retrieve(q'), holding the retriever, the index, the reranker and the generator fixed.

The deciding metric is recall@$k$ on a golden set that preserves the conversation. If you evaluate a rewrite against standalone questions you have removed the exact phenomenon the rewrite exists to fix, and it will look useless. Build the set from real history: the turn, the history, and the chunk that answers it.

Ablate one transformation at a time, on the same golden set and the same downstream stack:

The numbers are illustrative; the shape is what you look for. On a conversational benchmark, rewrite usually dominates; on an abstract single-turn benchmark, HyDE and multi-query dominate and rewrite is neutral.

Three measurements that separate real gains from vibes: a per-query-class breakdown (a technique that helps two classes and hurts one is a routing decision, not a switch); the conditional miss rate for multi-query — of the queries the original missed, what fraction did the union recover? Below ~20% (example) means the variants are too correlated to be worth the retrievals; and a no-transformation control on the transformed subpopulation, because a gain that also appears on untransformed queries is not caused by the transformation.

Watch for the metric moving for the wrong reason. Rewriting changes the query the generator sees too, if you pass the rewritten query into its prompt. Fix the generation prompt across arms, or you will attribute a prompt change to the retriever.

8. Interview drill

“A user asks a follow-up question and retrieval returns junk. Where do you look first?” Strong answer: the query itself. If the turn contains a pronoun or an elided noun phrase, the retriever is searching for a string that has no referent, and no index or embedder can fix that. Rewrite the query to be self-contained using the conversation history, constrained to substitution only. Common wrong answer: “the embedding model is weak” — swapping embedders will not recover a referent that was never in the query string.

“What does HyDE actually do, and why can a wrong hypothetical document still retrieve the right real one?” Strong answer: it replaces the query vector with the vector of an LLM-written hypothetical answer, so retrieval happens in answer space where the corpus lives. It survives fabrication because the encoder is a bottleneck: a pooled summary of a fluent passage preserves topic and entity structure while averaging out wrong specifics, and the ANN search then finds real documents in that neighbourhood. Common wrong answer: “it generates the answer” — it generates a retrieval key, and the paper is explicit that the document need not be true. Follow-up — “when does it hurt?”: when the query already carries the answer’s surface form (identifiers, codes, quoted strings), because hallucinating prose around them can move the vector away from the exact-match document; and where the generator lacks domain knowledge, since fluent wrong fiction gets rescaled rather than filtered. Deploy it routed to abstract natural-language questions, and measure per class.

“You add multi-query expansion with $m=4$ and recall barely moves. Why?” Strong answer: the variants are correlated. The union’s recall is $1-(1-p{)}^m$ only if misses are independent; LLM paraphrases share failure modes, so the conditional miss rate stays near 1. Diagnose by measuring that conditional rate, then diversify the generation (different aspects, not synonyms) or drop the technique. Common wrong answer: “increase $m$” — that multiplies latency against a saturated curve.

“Why is multi-hop retrieval not just a better retriever?” Strong answer: because the second fact is only findable given the first — “what to retrieve depends on what has already been derived”. No single retrieval call can formulate the second query, because the entity needed to formulate it is inside the first retrieved passage. Interleave retrieval with reasoning steps, or decompose up front if the sub-questions are genuinely independent. Common wrong answer: “retrieve more chunks” — useless if the second query was never issued.

“What is the cost, and when do you turn it on?” Strong answer: at least one extra LLM generation per transformation, several sequential ones for IRCoT. Against 40–60 ms of retrieval, that is often 10× the retrieval latency, and it lands on p99; the token bill is secondary. Turn it on per query class, after a golden set shows recall is the bottleneck and that class demonstrably benefits.

“How would you evaluate a rewrite step honestly?” Strong answer: on a golden set built from real conversations, with the raw turn as the baseline and retriever, reranker and generator fixed across arms; report recall@$k$ (the metric a rewrite can actually move) and end-to-end correctness, broken out by query class. Common wrong answer: evaluating on standalone questions, or using an LLM judge on the rewritten query’s fluency — neither measures whether the right chunk was retrieved.

Next: RAG Evaluation — every claim in this chapter about a transformation “helping” is a measurement claim, and that chapter is how you make it.