Chunking

TL;DR — A chunk is simultaneously the unit of retrieval, the unit of embedding, and the unit of citation, so the boundary you choose sets the recall ceiling for everything downstream. Two knobs decide most of it: chunk size $s$ and overlap $o$, with stride $d=s-o$. Overlap duplicates corpus tokens into the index — the cost multiplier is exactly $s/(s-o)$. A fact of length $b$ tokens is missed by every chunk with probability $(b-o)/(s-o)$ whenever $o<b\le s$, and is never retrievable once $b>s$. Those two formulas are how you pick a size without guessing: sample your answer spans, get $b$, then solve for the $(s,o)$ that hits your miss target at a cost you will pay.

1. Why this exists

Consider a runbook that ends with a support matrix, laid out as a table. A character-counting splitter cuts it at 1,000 characters — which lands in the middle of the table. The row that says gpt-4.1-mini | not supported ends up in chunk 47. The header row, model | status, ends up in chunk 46.

A user asks “is gpt-4.1-mini supported?” Chunk 47 contains the exact string and is retrieved. The generator receives a table fragment with no header, reads gpt-4.1-mini | not supported as… what? A column of model names? A filename? It answers, and it answers plausibly, and it is wrong. Nothing in the pipeline threw an error.

Three chunking failures are compressed into that one example:

1. The boundary destroyed the semantics of the unit. A table row without its header is not a fact; it is a fragment.
2. The retrieval unit is the citation unit. If the answer lives in chunk 47 but its meaning lives in chunk 46, the citation you show the user points at something that does not support the claim.
3. Nobody could see it. Retrieval returned a high-confidence hit. The failure only shows up when you read the chunk in isolation, the way the model does.

Now the opposite failure. Split the same runbook into 100-token windows so no table row is ever cut, and the sentence “the retention window is governed by the policy in the previous section” becomes its own chunk with no access to the previous section. Every chunk is a fragment, every embedding has too little signal, and top-$k$ fills up with near-identical pieces of the same paragraph.

Chunking is where a RAG system is quietly won or lost, and it is also the stage teams spend the least time on, because it feels like a preprocessing detail. It determines the best possible recall before a single model runs, and whether a correct answer can be cited.

The uncomfortable framing. Chunking is a lossy compression of your corpus into retrieval units: you are choosing, in advance and without knowing the questions, which co-occurrences of tokens survive as a unit. Overlap is the only insurance you can buy against that choice, and you pay for it twice — once in index size, once in duplicate context.

2. How it works

2.1 Six strategies and what each one is protecting

Only the recursive splitter is a robust default, precisely because it degrades gracefully: if it runs out of separators it emits an over-long chunk rather than crashing. That graceful degradation is also a trap — see §5.

flowchart TB
    A["Document"] --> B{"Reliable structure?<br/>(headings, HTML, code fences)"}
    B -- yes --> C["Structure-aware split;<br/>keep tables/code/functions whole"]
    B -- no --> D{"Prose with drifting topics?"}
    D -- yes --> E["Recursive split inside sections<br/>+ semantic boundary detection"]
    D -- no --> F["Recursive split;<br/>s from the sweep, o about 10-15%"]
    C --> G["Index the chunks"]
    E --> G
    F --> G

The last two branches of the table are the interesting ones, because they attack the real problem with chunking — not the boundary, but the fact that a chunk is embedded with no knowledge of its document. That is §2.4.

2.2 The overlap arithmetic, defined

Name the quantities once and use them throughout:

• $L$ — document length in tokens, not characters. This distinction is load-bearing; §3 shows a popular splitter that defaults to characters.
• $s$ — chunk size in tokens; $o$ — overlap in tokens, with $0\le o<s$.
• $d=s-o$ — the stride: the distance between consecutive chunk starts.
• $N=\lceil L/d\rceil$ — the number of chunks a document produces.
• $b$ — the length in tokens of the smallest span that answers a question. Call it the answer span. This is the number nobody measures and everybody needs.

Cost of overlap. The total number of tokens you embed is

$$T_{\text{embedded}}=N\cdot s\approx \frac{sL}{d}=L\cdot \frac{s}{s-o}$$

so the index inflation factor is exactly

$$\frac{T_{\text{embedded}}}{L}=\frac{s}{s-o}=\frac{1}{1-f},f:=\frac{o}{s}$$

where $f$ is the overlap fraction. The vector count is inflated by the same factor, so $f=0.10$ costs 11% more embeddings and 11% more index entries; $f=0.50$ costs precisely double.

What overlap actually buys. Model the answer span as starting at a uniformly random offset within a stride window — true if questions are unrelated to where you happened to cut. The span fits inside the chunk that starts at 0 exactly when

$$p+b\le s⟺p\le s-b.$$

Offsets in $(s-b,d)$ fail, an interval of length $d-(s-b)=b-o$. So

$$P(\text{span missed})=\{\begin{array}{ll}0,& b\le o\\ \frac{b-o}{s-o}=\frac{\beta -f}{1-f},& o<b\le s\\ 1,& b>s\end{array}\beta :=\frac{b}{s}$$

Every symbol here is doing work:

• $b$ — the answer span length. Without this number you cannot choose $s$ or $o$ rationally. It is measurable: sample 30 real questions, and count the tokens of the smallest span that contains the answer.
• $o$ in the numerator as a subtraction — overlap is a coverage guarantee for spans up to length $o$, and nothing more. “20% overlap means 20% fewer boundary problems” is wrong; 20% overlap guarantees correct handling of spans up to 20% of $s$ and gives partial coverage above that.
• $s-o=d$ in the denominator — a larger stride means fewer, wider-apart windows, so a given oversize span has more offsets at which it can fail. This is why raising $s$ while holding $f$ fixed reduces misses: $b-o$ and $d$ both scale with $s$, but $(b-o)/(s-o)$ falls when $b>o$.
• The $b>s$ case is not a probability, it is a certainty. A fact longer than the chunk size is unreachable by construction, however good the model is. That is the hardest constraint in the chapter.

Worked numbers at $s=512$:

Read the row for $b=200$ and notice what it costs to get the miss rate under 5%: $o$ must be at least 184 tokens, i.e. 36% overlap, a 1.56× index. Or you raise $s$: at $s=1024$ with $o=200$, a 200-token span is always contained ($b\le o$), for an inflation of $1024/824=1.24\times$ and — this is the counter-intuitive part — fewer chunks than the 512/20% configuration ($\lceil 10^6/824\rceil =1,214$ versus 2,440). Larger chunks with generous overlap are often cheaper than small chunks with token overlap, and the price you pay is embedding dilution, not index size.

That tension is the whole design problem: overlap protects spans; size protects the vector. There is no setting that is good at both, which is why §7 tells you to sweep instead of argue.

2.3 Why chunk size dilutes the vector

A single embedding is a point in ${\mathbb{R}}^d$. To be useful for retrieval, the model must place it near queries about the same topic. As $s$ grows, the encoder’s pooled representation becomes an average over more topics, and the cosine similarity to any particular query falls towards the corpus mean.

This is not a vague worry; it is the reason the two-stage retrieval design exists. At small $s$, individual chunks are sharply matched but the answer span is at risk of being cut. At large $s$, the answer span is safely inside, but the vector’s margin over unrelated chunks collapses — you get a long tail of chunks with scores near $0.75$, and top-1 becomes unstable across paraphrases. The observable difference: at small $s$ you see a few clear peaks and the failure mode is “the evidence was two chunks away”; at large $s$ you see many similar mid-range scores and the failure mode is “everything matched a little”. The fixes are asymmetric — more overlap and larger $s$ for the first, reranking and contextual headers for the second.

2.4 The two techniques that beat the boundary problem outright

Overlap duplicates text. These two instead give each chunk the context it lost, which is the better trade when you have the compute.

Late chunking (Günther et al. 2024, Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, arXiv:2409.04701). The normal order is chunk → embed, so each chunk is encoded in isolation. Late chunking inverts it: run the whole document through a long-context embedding model first, then apply the chunk boundaries after the transformer, “just before mean pooling” — hence “late”. Each pooled chunk vector is then a function of the full document’s token representations, so a bare “the company” in chunk 7 is resolved using text from chunk 1. It “works without additional training” and applies to any long-context embedding model. The cost: you lose cross-document batching, and the embedder’s window must cover the document.

Contextual Retrieval (Anthropic, 2024, engineering post). Prepend 50–100 tokens of LLM-generated, chunk-specific context before embedding and before BM25 indexing. The prompt hands the model the whole document plus the chunk and asks for “a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk.” So "The company's revenue grew by 3% over the previous quarter." becomes "This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was \$314 million. The company's revenue grew by 3% over the previous quarter."

Their measured results are why this technique spread: contextual embeddings alone reduced the top-20 retrieval failure rate from 5.7% to 3.7% (a 35% reduction); combined with contextual BM25, to 2.9% (49%); and with a reranker over the top-150 candidates down to top-20, to 1.9% (67%). The one-time indexing cost with prompt caching was $1.02 per million document tokens for 800-token chunks, 8k-token documents, a 50-token instruction, and 100 tokens of context per chunk. The caveat: because the cached prompt is the document, it is cheap only with prompt caching and consecutive processing of a document’s chunks.

3. Implementation

3.1 Real chunker configurations

# Deps: pip install "langchain-text-splitters>=0.3"
# Verified against langchain-text-splitters master [as of 2026-10]:
#   https://github.com/langchain-ai/langchain/blob/master/libs/text-splitters/
#     langchain_text_splitters/{base,character}.py
from langchain_text_splitters import RecursiveCharacterTextSplitter, TokenTextSplitter

# 1. The default everyone copies. RecursiveCharacterTextSplitter supplies
#    separators=["\n\n", "\n", " ", ""] and keep_separator=True, and inherits from
#    TextSplitter: chunk_size=4000, chunk_overlap=200, length_function=len.
#    The out-of-the-box unit is therefore CHARACTERS.
default = RecursiveCharacterTextSplitter(
    chunk_size=1000,      # characters, not tokens, unless you say otherwise
    chunk_overlap=200,    # f = 0.20 here
    add_start_index=True, # store the offset -> needed to build citations
)

# 2. What you should actually ship: count in the embedder's own tokens, so the
#    §2.2 formulas apply verbatim.
in_tokens = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    model_name="text-embedding-3-large",   # or pass a HF tokenizer instead
    chunk_size=512, chunk_overlap=64,      # tokens; f = 0.125
    add_start_index=True,
)

# 3. A literal stride of d = s - o, when you need exact arithmetic.
#    split_text walks token ids with start_idx += tokens_per_chunk - chunk_overlap.
#    No merge heuristics; raises ValueError if tokens_per_chunk <= chunk_overlap.
exact = TokenTextSplitter(model_name="text-embedding-3-large",
                          chunk_size=512, chunk_overlap=64)

Three behaviours worth knowing before you trust the recursive splitter:

• Its overlap is a target, not a stride. It merges pieces back together, popping from the front of the buffer until the accumulated length is at or below chunk_overlap. Realised overlap therefore varies chunk to chunk and is bounded by chunk_overlap, not equal to it. Use TokenTextSplitter when you need the $(b-o)/(s-o)$ arithmetic to hold exactly.
• Over-long chunks survive. If a piece is still longer than chunk_size and no separators remain, it is appended as-is, and _merge_splits logs "Created a chunk of size %d, ..." — a warning, not an error. The encoder then silently truncates it.
• Validation is minimal. chunk_size > 0, chunk_overlap >= 0, chunk_overlap <= chunk_size. The last allows chunk_overlap == chunk_size, i.e. a zero stride. Assert 0 <= o < s yourself.

3.2 Late chunking, and contextual headers

# Deps: pip install "transformers>=4.40"; late chunking: arXiv:2409.04701
# Needs a long-context embedder (e.g. BAAI/bge-m3, 8192-token input).
import torch
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-m3")
model = AutoModel.from_pretrained("BAAI/bge-m3").eval()

def late_chunk_embeddings(text: str, spans: list[tuple[int, int]]):
    """`spans` are token-offset ranges from any upstream chunker."""
    enc = tok(text, return_tensors="pt", truncation=True, max_length=8192)
    with torch.no_grad():
        hidden = model(**enc).last_hidden_state[0]        # (T, d)
    # Pool THIS span's token states. Every state came from a forward pass over the
    # whole document, so each pooled vector carries context from outside its span.
    # That is the entire method.
    return torch.stack([torch.nn.functional.normalize(hidden[a:b].mean(0), dim=-1)
                        for a, b in spans])

And the contextual-retrieval prompt, transcribed from Anthropic’s post:

<document>{{WHOLE_DOCUMENT}}</document>
Here is the chunk we want to situate within the whole document
<chunk>{{CHUNK_CONTENT}}</chunk>
Please give a short succinct context to situate this chunk within the overall
document for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.

3.3 The knobs

4. Numbers and trade-offs

The cost table you can compute from §2.2

For a 1M-token corpus, $\lceil 10^6/d\rceil$ chunks, one embedding each:

Two facts fall out of this table that contradict common intuition. First, the 1024/200 row is cheaper in both embeddings and vector count than the 512/36 row, while guaranteeing strictly better coverage of spans up to 200 tokens. Second, overlap is not the only lever; raising $s$ at fixed $f$ reduces the boundary miss rate too, because $(b-o)/(s-o)$ decreases in $s$ whenever $b>o$. Overlap is a guarantee; size is a probability.

What the index-time costs are

• Embedding chunks is one forward pass per chunk through a model 10–100× smaller than your generator: 2,440 chunks of 512 tokens is ~1.25M tokens of encoder work, minutes on one GPU, and the dominant cost of a re-index.
• Contextual Retrieval adds one LLM call per chunk: $1.02 per million document tokens with prompt caching and consecutive per-document processing. Without caching, the naive version re-sends the whole document per chunk — $O(L/d)$ full-document reads, roughly an 18× blow-up on the input side for an 8k-token document at $d=448$.
• Late chunking needs no extra passes but forbids cross-document batching.

Where chunking sits against the alternatives

Reranking cannot fix a chunk that never got retrieved, and a knowledge gap cannot be fixed by chunking. Ranked by cost-effectiveness for a recall-limited system:

The table is the honest answer to “should I spend the week on chunking or on the reranker?” Chunking moves the ceiling; reranking moves the ordering. If recall@100 is low, chunking and hybrid search are the levers. If recall@100 is high and recall@10 is low, chunking is not your problem.

5. Pitfalls

Measuring chunk size in characters while reasoning in tokens. $s=1000$ in English is roughly 250 tokens; in Chinese it may be near 1000 tokens; in code it is neither. Symptom: your formulas do not predict what you observe, and occasionally chunks get truncated by the encoder for no visible reason. Fix: from_tiktoken_encoder, or TextSplitter.from_huggingface_tokenizer with the embedder’s own tokenizer.

Over-long chunks that the recursive splitter emits on purpose. When no separator remains, the splitter appends the leftover piece unsplit and logs a warning. Symptom: a WARNING ... Created a chunk of size 1523, which is longer than the specified 512 in your logs that nobody read; questions about the tail of those chunks are never retrievable. Fix: assert max(len(c) for c in chunks) <= s after splitting, and fail loudly.

Cutting a table, a code block, or a numbered procedure. Symptom: the retrieved chunk is a fragment that reads as nonsense in isolation — the generator either refuses or fabricates. This is the failure in §1. Fix: structure-aware splitting; never split inside <table>, a fenced code block, or a list item; prepend the table header to every fragment if you must split a table.

Splitting the document away from its context. “The company’s revenue grew by 3%” with no indication of which company or which quarter. Symptom: questions that name the entity fail even though the fact is in the corpus. Diagnostic: print the top-5 chunks and read only those, with no other context — if you cannot tell what they refer to, neither can the embedder. Fix: contextual headers or late chunking.

Semantic chunking without a cost check. Semantic splitters embed every sentence before deciding boundaries, and the percentile threshold is dataset-dependent. Symptom: unmeasurable improvement, a doubled index build, and wildly variable chunk sizes. Fix: measure chunk-length variance and end-to-end recall before keeping it, and never tune the percentile on the questions you report on.

Chunk size larger than the embedder’s max_seq_length. Silent truncation at the encoder boundary, no error. Symptom: a bimodal failure pattern — questions answerable from the first half of a chunk work, those needing the second half never do. Fix: compare in tokens and enforce it.

Overlap that pollutes the prompt with duplicates. Two overlapping chunks both retrieved means the same paragraph appears twice in the context. Symptom: the generator repeats itself, or context precision drops for no obvious reason. Fix: deduplicate by source span before assembly.

Re-chunking without re-embedding. Changing $s$ or $o$ changes the chunk set and invalidates every vector. Symptom: the index holds vectors from two generations of chunking. Fix: put the chunking config in the index version key.

6. Using it in a project

Do not chunk at all if you do not have to

Same rule as The RAG Pipeline: if the whole corpus fits in the context window — under roughly 200k tokens (Anthropic) — chunking is pure downside, because there is no retrieval to get wrong. It earns its cost only when the corpus is larger than the window, or when you need access control finer than “the whole corpus”.

A decision procedure that does not require guessing

1. Measure $b$. Sample 30 real questions. For each, find the smallest span of source text that answers it and count its tokens. Take the 90th percentile, $b_{90}$. This single number resolves most of the design.
2. Pick $s\ge b_{90}$. If $b_{90}$ exceeds the embedder’s comfortable operating size, the answer is not one chunk: use a parent/child scheme (retrieve the small child, feed the generator its parent) rather than a bigger $s$.
3. Pick $o$ from the miss target. Solve $(b_{90}-o)/(s-o)\le ϵ$ for your $ϵ$; 5% is reasonable. If the required $o$ exceeds ~35% of $s$, raise $s$ instead — it beats raising $o$ for coverage and costs fewer vectors.
4. Choose structure over arithmetic where structure exists. If you have headings, split on them, then apply steps 2–3 only inside a section.
5. Sweep to confirm. On the golden set, sweep $s\in \{256,512,1024\}$ × $o\in \{0,10\%,20\%\}$ with a fixed embedder and retriever; report recall@$N$ and recall@10. Nine cells is a day’s work and beats any argument.
6. Add contextual headers or late chunking only after the sweep — they are the next tier and carry real index-time cost.

When not to invest in chunking

• recall@100 is already above ~0.95. Chunking is not your bottleneck; go to Retrieval: Hybrid Search and Reranking.
• Your documents are short and atomic. Tickets, chat messages, log lines, API reference entries: the natural unit is already the chunk.
• The failure is a knowledge gap, or the task is aggregate. No boundary choice creates a fact that is not in the corpus, and “how many customers churned in Q3?” is a text-to-SQL query, not a chunking problem.

7. How you would prove it worked

The measurement that matters

For a fixed embedder, fixed index, fixed $N$ and $k$, sweep the chunking configuration and report recall at both stages. The dependent variable is recall@$k$; the diagnostic variable is the rank distribution of the gold chunk.

# Pseudocode for the sweep. `golden` is [(question, gold_span_id)].
for s in (256, 512, 1024):
    for overlap_frac in (0.0, 0.10, 0.20, 0.36):
        idx = build_index(corpus, chunk_size=s, chunk_overlap=int(s * overlap_frac))
        ranks = []
        for question, gold_id in golden:
            ids = idx.search(question, k=N)       # retrieve DEEP (N, not k)
            ranks.append(ids.index(gold_id) + 1 if gold_id in ids else None)
        yield s, overlap_frac, recall_at(ranks, 100), recall_at(ranks, 10)

What the output looks like when chunking is the bottleneck (example):

(The recall numbers are illustrative; the shape — saturating recall as $s$ grows, and a further jump from contextual headers — is what you should expect to see.)

Notice what this table tells you that an accuracy number would not. Recall@100 rises monotonically with $s$ — larger chunks cope with longer answer spans — while recall@10 nearly plateaus after 512. That split is the “bigger chunks dilute the vector” effect: it appears as a growing gap between deep recall and shallow precision. It is why you measure recall@$N$ and recall@$k$ — the first diagnoses chunking, the second diagnoses ranking.

The boundary oracle

The direct test of the overlap model, and the one that catches “I set 20% overlap and nothing improved”. Take 100 questions with known answer spans. For each span, find every chunk that fully contains it; if none does, that is a hard boundary miss — unrecoverable by any retriever. Report the hard-miss rate and compare it against $(b-o)/(s-o)$ from your measured $b$ distribution.

If the observed rate is materially higher than predicted, your chunker is not honouring the stride you configured — that is the §3.1 “overlap is a target, not a stride” trap, and it is invisible in any end-to-end metric.

A/B that isolates chunking

To prove chunking (rather than the embedder or the reranker) caused a gain, hold everything else fixed and vary only the chunk set: same checkpoint, same normalisation, same $N$, same reranker, same prompt. Change the chunk size and the embedder in one release and you have learned nothing.

8. Interview drill

“How do you pick a chunk size?” Strong answer: do not pick it — measure. Estimate the answer-span length $b$ by sampling real questions and counting the tokens of the minimal supporting span. Require $s\ge b_{90}$ so long answers are representable at all, choose $o$ from the miss target $(b-o)/(s-o)\le ϵ$, then sweep $s\in \{256,512,1024\}$ against a golden set and verify with recall@100 and recall@10. If the required overlap exceeds ~35% of $s$, raise $s$ instead. Common wrong answer: “512 tokens with 10% overlap, that’s the standard.” It is a reasonable default and it is not a decision; it silently caps recall for any corpus with long answer spans.

“What does overlap actually buy, quantitatively?” Strong answer: with chunk size $s$ and overlap $o$, a fact of length $b$ is fully contained in at least one chunk with probability $1-(b-o)/(s-o)$ for $o<b\le s$; it is guaranteed for $b\le o$ and impossible for $b>s$. Overlap is therefore a coverage guarantee up to length $o$, not a percentage reduction in boundary risk. Its cost is an index inflation of exactly $s/(s-o)$. Common wrong answer: “overlap makes sure nothing falls between chunks.” It reduces the probability; only $o\ge b$ eliminates it, and $s\ge b$ is what makes elimination possible at all.

“Too-small chunks — what actually breaks?” Strong answer: two things. First, hard boundary misses: any answer span longer than $s$ becomes unreachable, and spans longer than $o$ are missed with probability $(b-o)/(s-o)$. Second, precision@$k$ degrades because $k$ slots fill with near-duplicate fragments of the same passage, and the citation unit becomes too small to support the claim. Follow-up: “and too-large?” → the pooled embedding becomes an average over several topics, so cosine similarity to any particular query is diluted. The symptom is a long tail of mid-range scores and unstable top-1. Fix it with a reranker and/or contextual headers rather than by shrinking $s$ and reintroducing boundary misses.

“How does late chunking differ from just using overlapping chunks?” Strong answer: overlap duplicates text into adjacent chunks but each chunk is still embedded in isolation, so cross-chunk context is not available to the encoder. Late chunking embeds the whole document first and applies the chunk boundaries after the transformer, just before pooling, so each chunk vector is a function of the whole document’s token states. It needs a long-context embedder and requires no training. Common wrong answer: “late chunking is overlap with extra steps.” Overlap changes what text is in the chunk; late chunking changes what the encoder could see when it produced the vector while leaving the text identical.

“How would you handle a corpus of tables?” Strong answer: never split inside a table. Detect the structure, keep the table whole if it fits, and if it does not, split by row while prepending the header row to every fragment. For numeric tables, route those queries to text-to-SQL — cosine similarity cannot do arithmetic or row filtering. Cite the table id plus the row id, so the citation points at a span that actually supports the claim.

“Your manager says to double the overlap because recall is low. What do you say?” Strong answer: overlap only helps when the miss rate is dominated by boundary effects. Measure the hard boundary-miss rate first — the fraction of answer spans that no chunk fully contains. If it is already below a few percent, doubling overlap doubles the index and changes nothing, and the low recall is coming from the embedder, the query prefix, or the absence of hybrid search. If it is high, the diagnosis is right and the fix is probably to raise $s$ (which also reduces chunk count) rather than to raise $o$. Common wrong answer: agreeing immediately. Overlap is the most over-applied knob in RAG, precisely because it is the one that sounds like it must help.

Next: Embedding Models — once the chunks exist, a single vector represents each one, and the choice of metric, normalisation, and dimensionality decides whether those vectors are comparable at all.