Memory and Context Management

TL;DR — The context window is a budget, not a container. Every turn re-sends the whole prefix, so cost grows quadratically in the number of steps and quality degrades as the window fills (“context rot”, “lost in the middle”). You manage it with four moves, in order of increasing violence: evict stale tool results, summarise (compact) the middle, externalise exact state into notes or a file store, and isolate sub-work into its own context. Never evict the original user intent; never rely on the model to remember what you threw away. A stable prefix is worth more than a short prefix, because prompt caching prices the stable part at 0.1x and any mutation at full price.

1. Why this exists

Consider the most ordinary agent task: “find why the nightly build broke and fix it.” The loop runs for 40 steps. Each step appends a thought, a tool call, and a tool result. A single read_file on a 2,000-line source file is roughly 25,000 tokens; a grep across a monorepo can be worse; a failing test log is 8,000.

The naive implementation is:

messages = [system, user]
while True:
    response = model(messages, tools)
    messages.append(response)
    if not response.tool_calls:
        break
    for call in response.tool_calls:
        messages.append(execute(call))

Nothing here ever removes anything. Three things then break, at different times:

The window overflows. At some step the request is rejected, or your provider silently truncates, or your own guard drops the oldest messages. If it drops the oldest messages it drops the system prompt and the user’s original request, and the agent starts confidently solving a different problem. That is the single most common way an agent “goes insane”.

Quality degrades before the window overflows. Chroma’s Context Rot report evaluated 18 models and found that “models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows” — even on simple tasks (Chroma, 2025-07-14). Anthropic’s framing is the same: context is “a finite resource with diminishing marginal returns”, an “attention budget” that “every new token … depletes” (Anthropic, Effective context engineering, 2025-09-29). Orthogonally, Lost in the Middle (Liu et al., arXiv:2307.03172) shows the beginning and end of a long context are attended better than the middle. So the budget is two-dimensional: how much, and where.

Cost grows faster than linearly. Every step re-sends the entire prefix, because the model is stateless. That is the arithmetic that forces the whole discipline, so derive it now rather than assert it.

Let the prefix entering step $i$ be $n_0+(i-1)\Delta$ tokens, where $n_0$ is the initial system prompt plus user request and $\Delta$ is the average tokens one step appends (thought + tool call + tool result). The total input tokens billed over $k$ steps is

$$C(k)=\sum _{i=1}^k(n_0+(i-1)\Delta )=k{n}_0+\Delta \frac{k(k-1)}{2}$$

• $k$ — the number of loop iterations. It is the variable you actually control: it is what a step cap bounds.
• $n_0$ — the fixed prefix. It contributes linearly: doubling it doubles cost. This is why tool inventories and verbose system prompts matter more than they look.
• $\Delta$ — the per-step growth. It contributes quadratically, through the triangular sum $\frac{k(k-1)}{2}$. This is the term compaction attacks: it resets $k$ to a small number while preserving the information.
• The sum is exact because the prefix at step $i$ contains every prior step’s output. Remove the append-only loop (say, by giving each step its own fresh context) and the quadratic term vanishes — which is exactly the sub-agent isolation strategy in section 6.

(example) With $n_0=4,000$, $\Delta =6,000$, and $k=40$: $C=160,000+4,680,000\approx 4.8$ million input tokens for one task. At $k=80$ the same workload costs roughly 19.2 million. Doubling the number of steps quadruples the bill.

Where this matters for you. In interviews, “context management” is asked as if it were about summarisation. It is not. It is about not growing the prefix, and summarisation is one of four tools.

2. How it works

2.1 The budget equation

Everything downstream depends on getting this one decomposition right:

$$T_{\text{avail}}=T_{\text{window}}-T_{\text{out}}-T_{\text{sys}}-T_{\text{tools}}$$

• $T_{\text{window}}$ — the model’s hard context limit in tokens. A property of the checkpoint, not of your application. Note that providers often advertise a larger window than the model uses well; section 4 covers that.
• $T_{\text{out}}$ — the tokens reserved for the model’s own response. This term exists because most APIs bill and enforce input and output against the same window. If you fill the window with input you have left the model no room to answer, and the request fails with a length error at the worst possible moment — after you have paid for the prefill. Reserve the maximum you will ever allow, not the average you expect.
• $T_{\text{sys}}$ — the system prompt: role, policy, output contract, and any few-shot exemplars.
• $T_{\text{tools}}$ — the serialised tool inventory: names, descriptions, and JSON Schemas. This term is the one people forget. It is charged on every request, it is invisible in the transcript, and with a large MCP tool surface it can be thousands of tokens (see MCP).

$T_{\text{avail}}$ is therefore the working space for conversation. Define occupancy

$${\rho }_t=\frac{n_t}{T_{\text{avail}}},\text{compact when }{\rho }_t\ge 1-ϵ$$

• $n_t$ — the tokens actually resident in the conversation at step $t$: all messages, including tool results.
• ${\rho }_t$ — the fraction of the working space consumed. It is the trigger signal, and a threshold, not a target. Running at $\rho =0.95$ is not “efficient use of the window”; it is a state where one 8,000-token tool result causes a hard failure.
• $ϵ$ — the headroom you keep for the single largest observation you are willing to accept. It should be sized from your worst tool, not chosen round: if read_file can return 25,000 tokens then $ϵ\ge 25,000/T_{\text{avail}}$. Set $ϵ=0$ and you get exactly the mid-task truncation failure. Set $ϵ=0.4$ and you compact constantly, paying a summarisation call every few steps.

2.2 Three tiers of memory, and why confusing them is the classic error

“Memory” in agent systems names three things with three lifetimes and three failure modes.

LangGraph draws exactly this line: checkpointers “persist a thread’s graph state as checkpoints … for short-term, thread-scoped memory”; stores “persist application-defined data outside the graph state … for long-term, cross-thread memory” (docs.langchain.com/oss/python/langgraph/persistence, [as of 2026-10]).

The working tier is the one that overflows. The other two are the ones that let you safely shrink it.

2.3 What to evict, and in what order

Eviction is not a sort by age; it is a sort by information density per token. Evict first — high volume, low residual value:

1. Raw tool results that are no longer referenced. This is the highest-value, lowest-risk eviction in the whole system. Anthropic calls clearing tool results “one of the safest lightest touch forms of compaction” and asks the right rhetorical question: “once a tool has been called deep in the message history, why would the agent need to see the raw result again?” Claude’s context editing “automatically clears stale tool calls and results from within the context window when approaching token limits” ([as of 2026-10]).
2. Redundant tool output — the same file read twice, the same search re-run. Detect by hashing the tool name plus canonicalised arguments.
3. Intermediate reasoning from completed subgoals. Summarise the conclusion; discard the deliberation that produced it.
4. Superseded versions of a file, plan, or diff. Keep the newest; keep the history in a store, not in context.

Evict last, and rename rather than drop:

5. Architectural decisions, unresolved bugs, implementation details. Claude Code’s compaction prompt “preserves architectural decisions, unresolved bugs, and implementation details while discarding redundant tool outputs or messages” ([as of 2026-10]). These are the facts whose loss makes a resumed agent repeat a mistake it already diagnosed.
6. Original user intent and constraints. Never evicted. Re-anchor: after any compaction, restate the task and its constraints verbatim at the top of the new context. This is not decoration — it is the one thing whose absence changes what the agent is solving.

Then, optionally, model it as a score. If you want automatic eviction rather than a rule list, the shape is

$$r(x)=\alpha \mathrm{recency}(x)+\beta \mathrm{refs}(x)+\gamma \mathrm{criticality}(x)-\delta \frac{\mathrm{size}(x)}{\bar{s}}$$

• $\mathrm{recency}(x)$ — steps since $x$ was created, decayed (e.g. $2^{-\Delta t/\tau }$). Without it you evict the step the agent is currently reasoning about.
• $\mathrm{refs}(x)$ — how many later messages reference $x$ (a file path, a tool_call_id). Without it you evict a result the agent has been citing for twenty turns. This term is the one that distinguishes eviction from truncation.
• $\mathrm{criticality}(x)$ — a small hand-assigned class score (user_intent > decision > observation > chatter). Without it, item 6 above gets evicted with everything else.
• $\mathrm{size}(x)/\bar{s}$ — token size normalised by the mean message size, subtracted. Without it you evict a 200-token decision to save room for a 12,000-token log dump.
• $\alpha ,\beta ,\gamma ,\delta$ — the exchange rates between these four quantities. They are the only tunables, and $\delta$ is the one that controls how aggressively you trade recall for space.

This scoring function is a design pattern, not a published algorithm. It is the shape every production evictor converges to; treat the weights as hyperparameters to fit on your own traces, not as defaults to copy.

2.4 Compaction, mechanically

Compaction is “taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary” (Anthropic, 2025-09-29). Concretely it is one extra model call:

$$c^{\prime }=\mathrm{LLM}\left(c_{[a,b]}\right)\text{then}{c}_{\text{new}}=c_{[0,a)}\Vert c^{\prime }\Vert c_{(b,k]}$$

• $c_{[a,b]}$ — the middle span being compressed. Not the whole history: the head ($c_{[0,a)}$) holds the system prompt and the user’s original request and is preserved verbatim; the tail ($c_{(b,k]}$) holds the last few turns, whose exact wording matters for continuity. Compressing the tail is what makes an agent forget what it was just doing.
• $c^{\prime }$ — the summary. It is itself context, so it must be budgeted: pick a target length $m$ and enforce it, or the summariser will return a 4,000-token “summary” of a 40,000-token span and you have paid for nothing.
• $\Vert$ — concatenation with an explicit delimiter and a provenance label, so the model can tell a summary from a live observation.

The tuning procedure is the most testable sentence in the literature:

“we recommend carefully tuning your compaction prompt on complex agent traces. Start by maximizing recall … then iterate to improve precision by eliminating superfluous content.” — Anthropic, Effective context engineering for AI agents, 2025-09-29

Read that as an experiment order: begin with a summariser that keeps everything possibly relevant, measure whether tasks still complete, then delete fields from the summary schema one at a time until completion drops. Starting from a tight summary optimises the wrong objective — you cannot tell which omission failed.

flowchart TD
    A["Step t: build request<br/>n_t tokens"] --> B{"rho >= 1 - eps ?"}
    B -- no --> C["Call model"]
    B -- yes --> D["Evict tier 1-4:<br/>stale tool results"]
    D --> E{"rho still >= 1 - eps ?"}
    E -- no --> C
    E -- yes --> F["Compact middle span<br/>c[a,b] -> c'"]
    F --> G["Re-anchor: prepend<br/>verbatim user intent"]
    G --> H["Invalidate prompt cache:<br/>full price next turn"]
    H --> C
    C --> I["Append observation"]
    I --> A

2.5 Externalise exact state; summarise narrative

Summaries are lossy in a specific, dangerous way: they lose numbers. Anthropic’s own example is an agent playing Pokemon that tracks “for the last 1,234 steps I’ve been training my Pokemon in Route 1, Pikachu has gained 8 levels toward the target of 10” and, “after context resets, … reads its own notes.” No summariser preserves 1,234 and 8 across a reset; a notes file does, because it is read back verbatim. The division of labour:

• Notes / a scratchpad file for exact state: counters, IDs, URLs, file paths, the current plan step, the last passing test name. Written by the agent, read back on demand.
• Compaction for narrative: what was tried, what was learned, why the approach changed.

Anthropic’s memory tool is the minimal durable tier: “Claude can create, read, update, and delete files in a dedicated memory directory stored in your infrastructure that persists across conversations… operates entirely client-side through tool calls.” Note what it is not: a file store exposed as tools, not a vector database. Retrieval is by naming a path, not by embedding similarity — deterministic, cheap, and inspectable.

Retrieval-backed memory (the RAG machinery plus write-back) is the right tool only when recall is genuinely fuzzy and cross-session (“what did we decide about the auth refactor three months ago?”). It inherits two costs the file store does not have: the store needs its own eviction policy, and the write path becomes an attack path — if the agent persists a memory from a context containing untrusted content, an attacker has installed a durable instruction. Treat memory writes as privileged operations (see Guardrails).

2.6 The cache interaction, which inverts the naive advice

Prompt caching prices tokens by position stability, not by count. Verified figures from Anthropic’s official cookbook ([as of 2026-10]):

• Cache writes cost 1.25x base input price; cache reads cost 0.1x.
• TTL: 5 minutes by default, refreshed on each hit, so an entry survives an active conversation; a 1-hour TTL is available at 2x base input price.
• Minimum cacheable length: 1,024 tokens for Sonnet; 4,096 for Opus and Haiku 4.5 — a prompt just under the threshold gets nothing.
• Up to 4 explicit breakpoints per request.

Now put a number on the inversion. Let $S$ be a stable cacheable prefix and $V_i$ the volatile tokens added at step $i$. Over $k$ steps, in units of the base input price:

$$C_{\text{cached}}(k)=1.25S+0.1S(k-1)+\sum _{i=1}^k{V}_i$$

• $1.25S$ — the one-time cache write. Paid once, on the first request that establishes the prefix.
• $0.1S(k-1)$ — the reads. This is the whole benefit: the stable part amortises to a tenth of list price.
• ${\sum }_i{V}_i$ — the volatile part, always at full price; nothing that changes between requests can be cached.
• Any mutation inside the prefix resets this. A timestamp, a random request ID, a re-ordered tool list, per-user text placed at the front — each one invalidates every token after it and you pay $1.25S$ again. This is why the MCP 2026-07-28 revision adds a SHOULD that servers “return tools from tools/list in a deterministic order to enable client-side caching and improve LLM prompt cache hit rates.”

Consequences that follow, each of which contradicts the obvious move:

1. A long stable prefix beats a short changing one. (example) At $S=20,000$ and $k=30$ the cached prefix costs $1.25\times 20,000+0.1\times 20,000\times 29=83,000$ price-units; the same tokens sent uncached 30 times cost $600,000$, and a 5,000-token prefix that changes every turn costs $150,000$ — nearly double the cached 20,000. Shorter is not cheaper. Stable is cheaper.
2. Compaction destroys the cache. Rewriting the prefix wholesale means the next turn pays full price for the new, shorter context plus a fresh write. Usually still a win, but it is a real cost, and it is why you compact in batches rather than continuously. The flip side is that a 5-minute human approval is cache-hostile on the default TTL — that is when the 1-hour TTL earns its 2x write price.

The one-line rule. Order your context from most stable to least stable: system prompt, then tool schemas in a deterministic order, then durable documents, then conversation, then the current observation. Anything that violates that order is paying full price for tokens that did not change.

3. Implementation

A real budget-and-compaction layer. Two dependencies, both pinned: anthropic 1.13.0 and tiktoken 0.14.0 ([as of 2026-10]).

# context_budget.py  —  anthropic 1.13.0, tiktoken 0.14.0, Python 3.10+
from dataclasses import dataclass, field

import tiktoken
from anthropic import Anthropic

MODEL = "claude-opus-5-5"          # model id, [as of 2026-10]
ENC = tiktoken.get_encoding("cl100k_base")   # local approximation; see note


def count_text(s: str) -> int:
    """Local token estimate: cheap, no network, approximately right.

    tiktoken is OpenAI's BPE, not Anthropic's tokenizer, so this drifts by a few
    percent. Use it for *policy* (when to compact); call
    client.messages.count_tokens(...) for *precision* near the limit.
    """
    return len(ENC.encode(s))


@dataclass
class ContextBudget:
    # T_window minus T_out: the space the provider will actually let you send.
    window: int
    # T_out: reserved for the model's reply. Reserve the MAX you will ever allow.
    reserved_output: int
    # T_sys + T_tools: fixed per-request overhead, measured once at startup.
    system_tokens: int
    # eps: headroom for the largest observation you are willing to accept.
    headroom: int
    # Target length of a compaction summary. Enforced, not requested politely.
    summary_target: int = 2000
    # Verbatim turns at the tail that compaction must not touch.
    keep_last_turns: int = 4
    messages: list[dict] = field(default_factory=list)

    @property
    def available(self) -> int:
        """T_avail from section 2.1."""
        return self.window - self.reserved_output - self.system_tokens

    @property
    def resident(self) -> int:
        return sum(count_text(_flatten(m)) for m in self.messages)

    @property
    def occupancy(self) -> float:
        return self.resident / self.available

    def needs_eviction(self) -> bool:
        return self.resident >= self.available - self.headroom


def _flatten(message: dict) -> str:
    """Stringify a message for counting. Tool results are the big ones."""
    content = message.get("content")
    if isinstance(content, str):
        return content
    parts = []
    for block in content or []:
        if isinstance(block, dict):
            parts.append(str(block.get("text") or block.get("content") or block))
    return "\n".join(parts)


def evict_stale_tool_results(budget: ContextBudget) -> int:
    """Eviction tiers 1-3. Returns tokens reclaimed.

    An observation is 'stale' when (a) it is not in the protected tail, and
    (b) no later message references its tool_call_id, and (c) it is not a
    user-intent or decision message. The reference check is what makes this
    safe to do automatically.
    """
    msgs = budget.messages
    tail_start = max(0, len(msgs) - budget.keep_last_turns * 2)
    later_text = " ".join(_flatten(m) for m in msgs[tail_start:])
    reclaimed = 0
    for i, m in enumerate(msgs[:tail_start]):
        if m.get("role") != "user" or not _is_tool_result(m):
            continue
        call_id = m.get("tool_call_id", "")
        if call_id and call_id in later_text:
            continue                      # still cited -> keep
        reclaimed += count_text(_flatten(m))
        msgs[i] = {"role": "user",
                   "content": f"[tool result {call_id} elided to save context]"}
    return reclaimed

Four things in that code carry the design:

• headroom is a size, not a percentage. Set it from the largest observation your tool surface can produce, because that is the thing whose arrival must not overflow the window. Percentage headroom silently fails when a tool’s output grows.
• reserved_output is charged against the same window. Treating it as free produces “max_tokens exceeds context limit” errors at the worst moment.
• The reference check (call_id in later_text) is the whole safety property. Without it, eviction is truncation with extra steps — and truncation loses the one result the agent is reasoning about.
• The replacement is a labelled placeholder, not deletion. Removing a message outright breaks the tool-call/tool-result pairing most chat APIs require (an assistant message with tool_calls must be followed by matching tool-role results). A tombstone keeps the transcript well-formed.

The compaction call itself:

COMPACT_PROMPT = """\
Summarise the conversation span below for an engineer who must continue the task.

Preserve, verbatim where possible:
- architectural decisions and the reason for each
- unresolved bugs, with the exact error text
- file paths, symbol names, IDs, and any number the task depends on
- what was already tried AND why it failed

Discard: raw tool output, redundant searches, superseded plans, deliberation.

Budget: {target} tokens. Output only the summary.

<span>
{span}
</span>
"""


def compact(budget: ContextBudget, client: Anthropic) -> ContextBudget:
    """Compress the middle span; preserve the head and the tail verbatim."""
    msgs = budget.messages
    # Head: system prompt + the user's original request and constraints.
    head, rest = msgs[:2], msgs[2:]
    # Tail: the last `keep_last_turns` user/assistant pairs.
    cut = max(0, len(rest) - budget.keep_last_turns * 2)
    middle, tail = rest[:cut], rest[cut:]

    span = "\n\n".join(_flatten(m) for m in middle)
    resp = client.messages.create(
        model=MODEL,
        max_tokens=budget.summary_target,     # hard cap = the budget, enforced
        messages=[{"role": "user", "content": COMPACT_PROMPT.format(
            target=budget.summary_target, span=span)}],
    )
    summary = "".join(b.text for b in resp.content if b.type == "text")

    budget.messages = head + [
        {"role": "user",
         "content": "<compact_summary>\n" + summary + "\n</compact_summary>"},
        # Re-anchor: restating intent costs ~50 tokens and prevents the failure
        # mode where the agent resumes on a different task.
        {"role": "user", "content": "<original_intent>\n" + _flatten(head[1])
          + "\n</original_intent>"},
    ] + tail
    return budget

Parameters worth naming, with the range you would actually use:

Do not call count_text in a loop over a large history. It is $O(n)$ re-tokenisation per step, turning a token problem into a CPU problem. Cache per-message counts and update only the messages you changed.

4. Numbers and trade-offs

The four strategies, costed

Notation: $n$ = tokens resident at compaction time, $m$ = summary length, $p_{\text{in}}$ = base input price per token, $p_{\text{out}}$ = base output price per token (typically 4-5x $p_{\text{in}}$).

The compaction call’s cost is

$$C_{\text{compact}}=n{p}_{\text{in}}+m{p}_{\text{out}}$$

• $n{p}_{\text{in}}$ — the prefill you pay to read the history in order to summarise it. This is why compaction is not free: you pay for the context once more, at full price, before you get to throw it away.
• $m{p}_{\text{out}}$ — the summary, at output price. Since $p_{\text{out}}>p_{\text{in}}$, a long summary is disproportionately expensive. This is the arithmetic behind “enforce summary_target”.
• There is no $k$ term, because this is a single call. That is the point: you are trading one large payment for the elimination of the quadratic term in $C(k)$ from section 1.

(example) At $n=150,000$, $m=2,000$, with $p_{\text{out}}=5p_{\text{in}}$: $C_{\text{compact}}=150,000+10,000=160,000$ input price-units. The 30 steps that follow then run against ~10,000 resident tokens instead of a growing prefix, which on the section-1 numbers saves an order of magnitude in the tail. Compaction pays for itself when the remaining task needs more than roughly $n/{\bar{n}}_{\text{turn}}$ further turns.

The cache interaction, quantified

Compaction and prompt caching pull in opposite directions, and you should measure both rather than pick a favourite. Compaction reduces the token count but rewrites the prefix, so the next few turns are all cache writes (1.25x); caching leaves a large prefix in place at 0.1x per read. Therefore: on a task with many short turns in a stable environment, do not compact at all — caching already gives you most of the saving and compaction destroys it. On a task where the prefix genuinely changes every turn (the agent writes files, tool output dominates), caching buys little and compaction is the only lever.

Vendor-reported effects, labelled as such

Anthropic’s context-management announcement reports, on their internal evaluations — not a controlled third-party benchmark — that a memory tool plus context editing improved performance “by 39% over baseline”, context editing alone “29%”, and that in “a 100-turn web search evaluation, context editing enabled agents to complete workflows that would otherwise fail due to context exhaustion — while reducing token consumption by 84%” (anthropic.com/news/context-management, 2025-09-29). Treat the direction as real and the magnitude as vendor-specific.

Which axis you are optimising

5. Pitfalls

Truncating the oldest messages. Symptom: the agent starts solving a plausible but different task, and the transcript no longer contains the user’s original request. Dropping from the front deletes the system prompt and the intent. Fix: evict by tier, and never evict the head.

Summarising the tail, or trusting the summary. Symptom (a): the agent asks a question it answered two turns ago, or re-runs a tool whose result is now inside the summary instead of in front of it. Symptom (b): figures drift — the agent reports “roughly 300 tests passing” when the number was 347, or cites a file path that was never mentioned. The last few turns must be verbatim, and anything numeric must be externalised rather than summarised.

Budget arithmetic errors. Two symptoms, one cause. (a) A hard length error at a random step that correlates with the largest tool output, not with step count — headroom was set to a percentage instead of to the size of your worst case. (b) You are consistently a few percent off the provider’s count and it only becomes visible near the limit. tiktoken is an approximation for non-OpenAI models; calibrate it against the provider’s own count_tokens endpoint.

Re-ordered tools or a timestamp in the system prompt. Symptom: cache hit rate near zero, cost several times your estimate, nothing visibly wrong. Anything that varies in the prefix invalidates everything after it. Sort tool schemas deterministically.

State that outlives the context is a liability. Symptom (a): notes exist on disk and are never retrieved, so the agent re-derives state it already wrote down — the read has to be part of the loop’s contract. Symptom (b): an assistant that behaves correctly for weeks then acts on an instruction nobody gave it, traceable to a page it summarised on Tuesday. The fix for (b) is to treat every memory write fed by untrusted content as a privileged operation.

Compaction storms. Symptom: wall-clock time dominated by summarisation calls, and each summary then immediately being summarised. The summary must be smaller than the span by a comfortable factor, or $\rho$ crosses the threshold again immediately.

6. Using it in a project

Reach for compaction when the work is genuinely long-horizon and conversational: a debugging session, an interactive refactor, a research thread where earlier findings inform later questions — the “extensive back-and-forth” case in the section 4 selection table.

Reach for a notes file when the state is exact and small: a todo list, a set of IDs, a running tally, the current plan step. This is the cheapest durable memory that exists and it is auditable by a human in a text editor.

Reach for sub-agent isolation when sub-tasks are independent and each produces a large intermediate that the parent does not need. The sub-agent works in its own context window and hands back only what it distilled: as Anthropic put it, subagents “facilitate compression by operating in parallel with their own context windows, exploring different aspects of the question simultaneously before condensing the most important tokens for the lead research agent.” Note that the published description gives no figure for how large that condensed summary is — treat the compression ratio as something you measure on your own task, not something you quote. This is the only one of the four strategies that makes the per-step growth $\Delta$ smaller rather than resetting it.

Reach for retrieval-backed long-term memory when recall is fuzzy and cross-session, and you can afford to own an eviction policy and a poisoning defence. If you cannot describe what a bad write looks like and how you would detect it, you are not ready for this tier.

When NOT to use any of this. Do not build a memory system for a two- or three-turn task. Do not add summarisation to an agent whose traces already fit comfortably in T_avail — you will pay an extra model call per task and lose fidelity for nothing. Do not build a vector-memory tier because it sounds like the mature thing to do; the honest ordering is eviction first (free), then notes (cheap), then compaction (one call), and only then retrieval. And do not assume a “long context” model removes the problem: Anthropic’s own caveat is that “context windows of all sizes will be subject to context pollution and information relevance concerns.”

Decision rule. Estimate $C(k)$ from section 1 with your measured $n_0$ and $\Delta$. Under about 20% of your per-task budget, stop — the problem you have is not context. Over it, the fix is almost always eviction plus sub-agent isolation, not a cleverer summariser.

7. How you would prove it worked

Three experiments, each with a baseline and an ablation. All numbers below are (example).

1. Task completion under a long-horizon workload. Metric: completion rate on a fixed suite of tasks that require more than one context window (the shape Anthropic uses is a 100-turn web search evaluation). Baseline: the naive append-only loop with truncation-from-the-front. Ablation: eviction only; eviction + compaction; all four. Report completion rate, p50/p95 wall-clock, and tokens consumed. The effect you want is completions that were previously impossible, not a small quality delta — a truncating baseline fails outright here, which makes the measurement clean.

2. Recall of the compaction summary, measured directly. Take 50 real traces, hand-label the facts a resumed agent must still know (paths, decisions, error strings, numbers), then measure recall $=\frac{|\text{facts in summary}|}{|\text{labelled facts}|}$ and precision as the fraction of summary content that is task-relevant. Section 2.4 says tune recall first and precision second, so plot both and report where you stopped.

3. The cache-vs-compaction cost ablation. Run (a) no compaction with a stable prefix, (b) compaction with a stable prefix, (c) no compaction with a mutated prefix; measure total input tokens and cache hit rate. This stops you compacting a workload that caching already made cheap, and catches a prefix mutation that is costing you 5x.

Reporting template (example). “On the 60-task long-horizon suite the append-only baseline completed 41%; tiered eviction alone reached 63%; adding compaction reached 78%. p95 latency fell from 11m to 7m because truncation failures cost a retry. Compaction added one call per task, ~6% of task tokens. Recall-first summary tuning beat precision-first by 4 completion points and cost 12% more tokens; we kept recall-first.”

8. Interview drill

“The conversation exceeds the window. What do you do?” Strong answer: in order — evict unreferenced tool results (free), summarise the middle while preserving the head and the last few turns verbatim, externalise exact state into notes, and isolate independent sub-work into its own context window. Name what must never be dropped: the original user intent and constraints, architectural decisions, unresolved bugs, and identifiers. Common wrong answer: “summarise the conversation.” That is one of four moves, it is the most expensive one, and it is the one that loses exact numbers.

“Why not just use a bigger context window?” Strong answer: because quality degrades with length before the limit is reached — context rot is measured across 18 models, and lost-in-the-middle shows position matters — and because cost is quadratic in steps regardless of window size. A bigger window moves the failure later; it does not change the shape. Common wrong answer: “bigger windows cost more per token.” True but tangential; the load-bearing point is the quality gradient and the step arithmetic.

“What is the cost of compaction?” Strong answer: one full-context call: $n{p}_{\text{in}}$ to read the span plus $m{p}_{\text{out}}$ for the summary, plus the loss of the prompt cache because the prefix is rewritten. It is worth it when the remaining task needs enough turns that the quadratic term in the append-only cost dominates. Follow-up: “how do you choose the summary length?” → enforce a token target; the summary is context too, and an unbounded summariser defeats the purpose.

**“How does prompt caching change your context strategy?”**Strong answer: caching prices stability, not size — writes at 1.25x, reads at 0.1x. So a long stable prefix is cheaper than a short volatile one, and anything that mutates the prefix (a timestamp, re-ordered tools) costs you the whole cache. Compaction and caching are in tension: compacting rewrites the prefix. Common wrong answer: “caching just makes everything cheaper.” It makes stable things cheaper and leaves volatile tokens at full price.

“A summary said 300 tests passed but the real number was 347. How do you prevent that?” Strong answer: the failure is category-level, not prompt-level — summaries paraphrase and drop numbers. Externalise exact state: have the agent write counters, IDs, and paths to a notes file or memory tool, and read them back verbatim. Also constrain the summary schema to fields you have ablated. Common wrong answer: “add ‘preserve all numbers’ to the summarisation prompt.” It helps at the margin and fails under load, because the summariser has no way to know which number matters.

“What is the difference between short-term and long-term memory in an agent?” Strong answer: short-term is thread-scoped state persisted so a conversation can resume — a checkpointer keyed by thread_id. Long-term is cross-thread application data — a store or file system. Different mechanisms, different lifetimes, and different trust: long-term memory is a durable attack surface if the agent can write to it from untrusted content. Trap: describing them as “the same thing with different TTLs.”

Next: MCP (Model Context Protocol) — the tool inventory you must budget for is what MCP standardises, and its 2026 revision changed how it is negotiated.