AI-Assisted Coding

TL;DR — A coding assistant is a generator plus a verifier, and the verifier is the part that decides whether the output is usable. pass@k is a capability metric: it measures whether a correct solution exists among $k$ samples, and you pay $k$ times the verification cost to find it. Without an independent verifier — tests the model did not write, a type checker, a human who understands the domain — more samples buy variance, not correctness. HumanEval-style benchmarks saturate and mislead: augmenting HumanEval’s tests 80-fold dropped measured pass@k by 19.3-28.9% relative and reordered the leaderboard. The sanity rule is: only let a model edit what you can check, and the cost that decides whether it pays is review throughput, not generation cost.

1. Why this exists

A model produces a 40-line patch that adds a cache to a hot path. It compiles. The existing tests pass. It is reviewed in four minutes — the code reads well, which is exactly the property a language model is best at producing — and merged. Three weeks later, under concurrent load, two requests race on the same key and one of them serves stale data to the wrong tenant.

Nothing in that story is unusual, and the interesting part is which control failed:

• Not generation. The model did what it was asked.
• Not compilation. It compiled.
• Not the test suite as written. The suite had no concurrency test, because the suite was written before anyone thought about the cache.
• Not even the reviewer, exactly. The reviewer had four minutes because there were eleven other patches that day, and plausible-looking code is expensive to distrust.

So the failure is structural: generation got cheaper and verification did not. When a model can produce a hundred plausible diffs an hour and a human can carefully review four, the binding constraint moves to review, and the review quality is what your defect rate now tracks. Everything in this chapter follows from that.

Three problems must be solved in that order, each with its own metric:

1. Context selection. Does the model see the code it must change, and enough of the surrounding contract? If not, sampling does not help — this is a recall ceiling, the same shape as schema linking in NL2SQL.
2. Edit expression. Can it express the change in a form your tooling applies reliably? This is an engineering problem, not a model problem, and it is where most “the assistant is flaky” complaints live.
3. Verification. Does anything independent check that the change does what the issue asked? This is the only step that catches a plausible wrong answer, and the step that costs human attention.

2. How it works

2.1 The loop

flowchart TD
    I["issue / instruction"] --> R["context selection:<br/>ranking, retrieval, repo map"]
    R --> P["propose edit<br/>(whole file, diff, or search/replace)"]
    P --> A["apply in a scratch checkout"]
    A -->|does not apply| P
    A --> S["static checks:<br/>compile, lint, typecheck"]
    S -->|fail| FB["failure text"]
    S --> T["run the test suite"]
    T -->|fail| FB
    T -->|pass| D["diff review<br/>(human, on the change only)"]
    FB --> P
    D -->|reject| P
    D -->|accept| M["merge in a sandboxed PR"]
    M --> CI["CI + canary in production"]
    CI -->|regression| RB["revert"]

Two properties of that diagram matter.

Every loop edge is a cost multiplier, not a quality guarantee. A retry round re-embeds the repository context and re-prefills the prompt. An agent with $h$ hops pays $h$ prefill latencies instead of one, which is why prompt caching and context trimming matter more here than in single-shot completion (Playbook, §4.2).

The accept edge is the only place correctness is assessed, and it is a human. Tests, type checkers and linters can reject; only a reviewer can reject something that passes them. Design the loop so those minutes go to intent, data model and failure semantics, and automate the rest.

2.2 pass@k measures existence, not delivery

The standard code-generation metric is pass@k: given $k$ samples per problem, is at least one correct? The unbiased estimator from the Codex paper (Chen et al., arXiv:2107.03374) is

$$pass@k={\mathbb{E}}_{\text{problems}}\left[1-\frac{\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)}{\left(\genfrac{}{}{0ex}{}{n}{k}\right)}\right]$$

• $n$ — how many samples were actually generated per problem. It must be at least $k$, and the estimator’s variance falls as $n$ grows.
• $c$ — how many of those $n$ passed the tests. This is the only signal: the estimator uses the count of successes, not which ones.
• $k$ — the number of samples you are allowed to submit. The binomial $\left(\genfrac{}{}{0ex}{}{n}{k}\right)$ counts all $k$-subsets, and $\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)$ counts the subsets drawn entirely from the failures. Their ratio is the probability that a random $k$-subset happens to contain no correct sample, so one minus that is the probability it contains at least one.
• The expectation over problems is a plain mean over the evaluation set.

Why not the obvious $1-(1-c/n{)}^k$? Because it assumes sampling with replacement and mis-estimates badly at small $n$, which is where code benchmarks live. With $n=5$, $c=1$, $k=5$ the truth is certain — you have five samples and one of them passed, so a $k=5$ submission always contains a passing sample — and the unbiased estimator returns exactly 1.0. The naive formula returns 0.672:

The gap closes as $n$ grows, which is why the bias is invisible on a 200-sample sweep and fatal on a five-sample one:

The trap is reading pass@100 as a product claim. pass@100 = 0.70 does not mean the assistant solves 70% of your tickets; it means that if you generate 100 candidates and can find the right one, one exists. Finding it is the verifier, so the product metric is not pass@k at all — it is pass@1 conditional on a verifier. That is why the next subsection matters more.

2.3 The verifier is the real bottleneck

Let the verifier be the test suite plus static checks, and characterise it by two numbers:

• $\alpha$ — the false-accept rate: the probability that a wrong patch passes the verifier. This is a property of your test suite’s coverage, and it is the number the benchmark literature actually measures.
• $\beta$ — the false-reject rate: the probability that a correct patch fails the verifier. Flaky tests, environment drift, snapshot files, and tests that over-specify implementation all push this up, and every unit of $\beta$ is paid in retries and human time.

If you sample $k$ candidates and take the first that passes, the quality of what you merge is bounded by $\alpha$, not by $k$:

$$P(\text{accepted patch is wrong})\approx \frac{\alpha (1-p)}{\alpha (1-p)+(1-\beta )p}$$

where $p$ is the per-attempt probability that the patch is genuinely correct. The formula’s shape is the point: as $k$ grows you get more shots at passing the verifier, so the accepted set fills up with patches that satisfy the tests — and if the tests are the thing that is wrong, you have manufactured a pipeline for producing confidently wrong code. Sampling more does not fix a bad verifier; it exploits it.

This is measured, not hypothesised. EvalPlus (arXiv:2305.01210) extended HumanEval’s test cases by 80x using automatic input generation and found that measured pass@k fell by 19.3-28.9% relative across 26 models. More damning than the drop: the ranking changed. Models that appeared to beat ChatGPT on HumanEval stopped beating it on HumanEval+, so the benchmark’s ordering was an artifact of test insufficiency, not of capability.

The same $\alpha$-failure has a domestic form: the model writes the tests. A patch plus its own tests is a self-consistent pair, and the tests encode the implementation’s assumptions rather than the requirement — a green build over behaviour nobody specified. The fix is procedural: tests come from the issue’s acceptance criteria, from a different author, or from a mutation-testing pass that checks the suite could have caught a deliberately introduced bug.

2.4 Context selection is also a recall problem

Let $F^{\ast }$ be the set of files the correct patch must touch, and $F$ the files the retriever put in the prompt:

$$\mathrm{ctx}\text{-}\mathrm{recall}=\frac{|F\cap F^{\ast }|}{|F^{\ast }|},\text{tokens}\approx \gamma |F|+\delta$$

• $F^{\ast }$ — the required files. On your own repository you can compute this offline from historical commits, which makes it measurable rather than aspirational.
• $F$ — what the model actually sees. A file the model never sees is a file it cannot correctly edit, so a missing member of $F^{\ast }$ is a hard ceiling on the patch’s correctness, exactly like a missing schema column in NL2SQL. The model will still produce a patch; it will simply be wrong in a way that compiles.
• $\gamma$ — tokens per file once rendered; $\delta$ — the fixed prompt (system instructions, issue text, output format). Retrieval raises $|F|$ monotonically and lowers precision monotonically, so the operating point is chosen by where the marginal recall gain goes to zero, not by “retrieve more”.
• The cost is superlinear in practice even though the token count is linear, because attention degrades over long contexts: performance is highest when the relevant document sits at the beginning or end of the context and drops when it is in the middle (Lost in the Middle, arXiv:2307.03172). Two prompts with identical token counts can therefore have different effective recall.

The practical corollary: rank, do not just retrieve. For a coding agent, the ranking features that matter are cheap and structural — is the file imported by the failing stack trace, does it define a symbol the issue names, was it touched by the last five commits to this subsystem — rather than embedding similarity to the issue text.

2.5 Edit format is an engineering choice with a measured effect

Three formats, in increasing order of how brittle they are:

The SWE-agent result is that this is not cosmetic: giving the agent a purpose-built agent-computer interface — constrained edit commands, a repository viewer, a linted output format — raised pass@1 on SWE-bench to 12.5% and on HumanEvalFix to 87.7%, both state of the art at the time, with the paper’s own conclusion being that interface design drives agent behaviour. If your assistant fails to apply edits, the fix is usually a stricter edit format rather than a better model.

3. Implementation

The listing below is a verification loop: propose an edit, apply it to a fresh scratch checkout, run the suite, feed the failure back, and stop on the first accepted attempt. The model is injected, so the loop is testable without a network call. Python 3.11+, standard library only.

"""A verification loop for model-authored edits, plus the pass@k estimator."""
import math
import os
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from typing import Callable


def pass_at_k(n: int, c: int, k: int) -> float:
    """Unbiased pass@k estimator (Chen et al. 2021, arXiv:2107.03374, eq. 1).

    n = samples generated per problem, c = how many passed, k = how many you
    may submit. c/n is unbiased for k=1 only; 1-(1-c/n)**k is biased at small n.
    """
    if k > n:
        raise ValueError("k must be <= n")
    if n - c < k:          # fewer than k failures: some k-subset always passes
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)


@dataclass
class Attempt:
    patch: str
    accepted: bool
    output: str


def run_tests(workdir: str, timeout_s: float = 30.0) -> tuple[bool, str]:
    """The verifier. In production this is the project's real suite in CI."""
    proc = subprocess.run(
        [sys.executable, "-m", "unittest", "discover", "-s", ".", "-q"],
        cwd=workdir, capture_output=True, text=True, timeout=timeout_s,
    )
    return proc.returncode == 0, (proc.stdout + proc.stderr).strip()


def verify_loop(project: dict[str, str],
                propose: Callable[[str | None], str],
                max_rounds: int = 3) -> list[Attempt]:
    """Propose -> apply in a scratch copy -> run tests -> feed failure back.

    Each round starts from a *fresh* copy of the project. Reusing a dirty
    worktree is the most common way to get a green result that does not
    reproduce, because a rejected edit leaves half of a refactor behind.
    """
    attempts: list[Attempt] = []
    feedback: str | None = None
    for _ in range(max_rounds):
        patch = propose(feedback)
        workdir = tempfile.mkdtemp(prefix="verify-")
        try:
            for rel, body in project.items():
                path = os.path.join(workdir, rel)
                os.makedirs(os.path.dirname(path), exist_ok=True)
                with open(path, "w", encoding="utf-8") as fh:
                    fh.write(body)
            with open(os.path.join(workdir, "calc.py"), "w", encoding="utf-8") as fh:
                fh.write(patch)
            ok, output = run_tests(workdir)
        finally:
            shutil.rmtree(workdir, ignore_errors=True)
        attempts.append(Attempt(patch, ok, output))
        if ok:
            return attempts
        feedback = output[-2000:]      # the failure, not the whole log
    return attempts

Driving it with a project whose calc.py is wrong and a scripted model that first proposes the same wrong code, then uses the failure text:

# continuing from the listing above
PROJECT = {
    "calc.py": "def add(a, b):\n    return a - b\n",
    "test_calc.py": (
        "import unittest\nfrom calc import add\n\n"
        "class TestAdd(unittest.TestCase):\n"
        "    def test_two_positives(self):\n        self.assertEqual(add(2, 3), 5)\n"
        "    def test_negative(self):\n        self.assertEqual(add(-1, 1), 0)\n"
    ),
}
WRONG = "def add(a, b):\n    return a - b\n"
RIGHT = "def add(a, b):\n    return a + b\n"

def scripted_model(feedback):
    if feedback is None:
        return WRONG                      # round 1: no improvement
    assert "AssertionError" in feedback    # the verifier told us what broke
    return RIGHT                          # round 2: uses that information

attempts = verify_loop(PROJECT, scripted_model, max_rounds=3)
for i, a in enumerate(attempts, 1):
    print(f"round {i}: accepted={a.accepted}")
print("rounds used:", len(attempts))

Which prints:

round 1: accepted=False
round 2: accepted=True
rounds used: 2

Three details fix specific observed failures: a fresh copy per round, because a dirty worktree lets a rejected attempt’s partial refactor make the next run pass for a reason that will not reproduce; feedback truncated to the tail, since the full log spends context on stack frames the model cannot use while “FAILED” alone gives it nothing to act on; and bounded rounds, a cost control rather than a quality one, because more rounds do not fix a verifier that accepts wrong patches (§2.3).

The knobs that matter

4. Numbers and trade-offs

The benchmarks, and what each one actually measures

Four things about that table that should change how you quote a number:

1. The 1.96% → ~79% jump is mostly scaffold, not model. On SWE-bench Verified [as of 2026-10], live-SWE-agent + Claude Opus 4.5 resolves 396/500 (79.2%) and OpenHands + Claude Opus 4.5 resolves 388/500 (77.6%), while the bash-only comparison harness (mini-SWE-agent v2.4.2 with a minimal ReAct loop and no special tools) gets 359/500 (71.8%) with Gemini 3.5 Flash — and that run produced no output at all on 59 of the 500 instances. Quote the harness with the number, always.
2. Leaderboard entries are not like-for-like. SWE-bench Verified’s own documentation notes that a release change from parsing action strings to tool calling makes results across major versions not comparable, so the leaderboard scores “release 1.x” and “2.x” results that are not the same experiment.
3. Verification is where the biggest measured effect lives. EvalPlus’s 80x test augmentation changing both scores and ranking shows the verifier, not the generator, was the limiting factor.
4. Saturation destroys discrimination, not difficulty. HumanEval at 164 problems was a fine probe in 2021 and is a coin flip now.

Where the cost actually goes

Derive it rather than asserting it. Take an agent that makes $h$ hops, each re-sending a context of $T$ tokens (system prompt, repo context, trailing history), with a cached prefix fraction $\rho$:

$$\text{prefill tokens}\approx hT(1-\rho )+T$$

• $h$ — model calls in the loop: one for a completion, five to thirty for an agent that reads files, runs tests and retries.
• $T$ — prompt tokens per hop. It grows with $|F|$ from §2.4, so context selection and cost are one decision seen from two sides.
• $\rho \in [0,1]$ — the fraction of the prompt that is a stable prefix served from cache. This is the term that makes prompt caching a first-class requirement for coding agents rather than an optimisation: at $\rho =0.9$ and $h=10$, the same $T$ costs roughly a tenth of the uncached prefill.
• The trailing $+T$ counts the final call, which is unavoidable.

With $T=30,000$ tokens (example: a moderately sized repo slice), $h=10$, and no caching, that is 300,000 prefill tokens to land one edit. Caching the stable 90% and trimming the slice to 12,000 turns the same workflow into roughly $10\times 12,000\times 0.1+12,000=24,000$ tokens — a factor of 12 on the dominant term. That is why “which files do we put in the prompt” is a budget question even before it is a correctness question.

5. Pitfalls

Benchmark score as a proxy for your repository. Symptom: an assistant that tops SWE-bench Verified cannot fix a bug in your monorepo, and its failures are all context, not reasoning — it edits the wrong package, or invents an internal API with a plausible name. SWE-bench repositories are public Python projects with runnable suites; your repository is a different distribution. Fix: build an internal eval set from your own closed issues (100-300 is plenty), and report that number, not the vendor’s.

Tests authored by the same model. Symptom: the build is green and the behaviour is wrong, and the test reads like a restatement of the implementation (assert result == compute_expected(...), with compute_expected defined in the same patch). This is a false-accept-rate $\alpha$ failure, and no amount of resampling fixes it. Fix: acceptance criteria from the issue, independent test authorship, or mutation testing to measure whether the suite could fail.

Context overload. Symptom: the model edits a file that does not contain the bug, or imports a function that does not exist. Both are recall-and-precision failures of §2.4: too little context to see the real definition, too much context to attend to it. Fix: rank by structural signals (stack trace, symbol definitions, recent commits) rather than embedding similarity, and measure ctx-recall against historical commits.

Silent scope creep in a “refactor”. Symptom: a diff that fixes the bug and also renames two functions, reformats a third file, and changes a public signature without updating callers — and passes, because the callers were not in the test path. Fix: read the diff stat before the diff, and reject changes touching files outside the issue’s blast radius.

Hallucinated dependencies. Symptom: an import of a package that does not exist, a real package used through an invented API, or an install instruction for a name a third party can later register. Fix: allow lists, a lockfile check in CI, and never auto-install what a model suggests.

Review throughput collapse. Symptom: PRs per engineer double, review latency triples, defect escapes rise — the signature of a generation/verification mismatch. Fix: cap model-authored diffs in flight per reviewer and shrink diffs until review depth is preserved. This is the pitfall that turns a productivity win into a quality regression, and it is invisible to any metric that counts output.

Over-trusting an explanation. Symptom: a confident paragraph explaining why the patch is correct, accepted instead of the diff. The prose comes from the same process as the bug. Fix: review the diff; a description that contradicts it is a signal, not a summary.

6. Using it in a project

Where it pays, by return per unit of trust: tests written against a specification you have; mechanical refactors where the compiler is the verifier; explanations and navigation (“what reads this config key?”); error triage from a stack trace; boilerplate from a schema or template.

Where it must not be trusted, whatever the benchmark says: authentication and authorisation, cryptography, concurrency, financial arithmetic, migrations with no rollback, and anything that mutates production state. The criterion is not difficulty but blast radius times verification cost: if a wrong answer is undetectable by your suite and expensive to reverse, a model is the wrong author.

When NOT to use it at all:

• When there is no verifier. If the change cannot be checked by a test, a type, a linter, or a domain-aware human in a few minutes, a model author adds a defect channel. Build the verifier first; that is the real work.
• When a deterministic tool does it. A formatter, codemod, sed, type-aware refactor or schema-driven generator is exact and fails read-only. Rung 1 on the decision ladder.
• When the diff is smaller than the prompt. One changed line behind a thousand tokens of context is a slow, lossy way to type.
• When review is the constraint. A team that cannot review one more diff this week does not gain throughput from a new author; it gains a queue.

Review process, concretely. Accept a diff when it touches only files inside the issue’s blast radius; it compiles and type-checks in a sandbox; the tests covering the changed behaviour were not written by the same patch; the diff stat fits the reviewer’s time budget; and the author can name the failure mode the change is exposed to.

7. How you would prove it worked

The trap is measuring generation speed. The metric that matters is downstream.

Primary metric: first-pass acceptance rate on model-authored diffs — the fraction merged without substantive review changes — paired with defect-escape rate (post-merge bugs attributed to those diffs). One without the other is trivially gameable: acceptance rises if reviewers get tired, and defect rate falls if the diffs are tiny.

Secondary metrics, each diagnostic of something different:

• Review latency p50/p95, model-authored vs human-authored. If it rises, you moved the bottleneck instead of removing it.
• Revert rate within 14 days. High revert rate with high acceptance is the signature of a weak verifier.
• Test provenance ratio: the fraction of a change’s tests written by the same patch. Report it as a percentage, not a policy.
• Internal eval score split by task type (bug fix / feature / refactor / test authoring). The aggregate hides that assistants are strong on one and weak on another.

Baseline: the same team and repository before the tool, same task types, or a control repository. “We shipped more PRs” is not a result without the defect rate that came with them.

Ablations: repo-context retrieval on/off at fixed model (tests §2.4); $k\in \{1,3,10\}$ candidates with and without a selective verifier (tests §2.3 — the prediction is that $k$ helps only if the verifier can rank, not merely accept); search/replace vs patch (measures apply rate, not quality); and test-authorship restriction on/off, the highest-value one because it targets $\alpha$.

Reporting template (example — replace every number). “Over six weeks on one service, 214 model-authored diffs had a 71% first-pass acceptance rate at a median review latency of 26 minutes, against 88% and 41 minutes for human-authored diffs of comparable size. Defect-escape rate was 2.8 per 100 diffs versus 1.9 for human-authored work — the change was faster, not yet better. Restricting tests to independent authorship and raising the internal context recall from 0.72 to 0.94 took acceptance to 83% and escapes to 1.6. What still fails: cross-package changes where the agent does not know a shared interface exists; the next fix is a repo-level symbol index, not a larger model.”

That report is credible because it names the baseline, reports the metric that could have contradicted the story, and attributes the residual failures to a layer.

8. Interview drill

“How would you evaluate a coding assistant?” Strong answer: two levels. Offline: an internal eval set from your own closed issues, split by task type, scored by tests the model did not write, plus first-pass acceptance. Online: review latency, revert rate and defect-escape rate against a human-authored baseline. Common wrong answer: “we’d look at SWE-bench Verified.” That measures a capability on public Python repositories with runnable suites, and it is heavily scaffold-dependent — the same model scores very differently under a bash-only harness and under an agent that reads files and runs tests.

“What does pass@k actually measure?” Strong answer: whether a correct sample exists among $k$, estimated unbiasedly by $1-\left(\genfrac{}{}{0ex}{}{n-c}{k}\right)/\left(\genfrac{}{}{0ex}{}{n}{k}\right)$; it is a capability metric, not a delivery metric, because finding the correct sample requires a verifier and costs $k$ times the checking. Common wrong answer: “the probability it gets the answer right if you ask $k$ times”, computed as $1-(1-p{)}^k$. That is a different (biased, small-$n$) estimator, and it hides the fact that you must still identify the right one.

“The tests pass. Is the patch correct?” Strong answer: only as far as your suite’s false-accept rate $\alpha$ is low — and that is measurable: EvalPlus’s 80x test augmentation cut measured pass@k by 19.3-28.9% and changed the leaderboard, so test insufficiency is first-order even on curated benchmarks. The sharpest version: if the same patch authored the tests, the verifier is not independent. Trap: “the tests pass, so yes.” That answer ships green builds with wrong behaviour.

“Where would you not use an AI coding assistant?” Strong answer: where blast radius times verification cost is high — authentication, crypto, concurrency, destructive migrations — and where no verifier exists. Also where a deterministic transformation does the job: a codemod or formatter is exact and fails read-only; a model is neither. Common wrong answer: “anywhere, with careful review.” Review is the constraint the tool strains; “with review” answers a different question.

“You have a failing test and an agent that can retry. How many rounds?” Strong answer: one or two. Each round re-pays the prefill, and the acceptance criterion is the verifier, so extra rounds only add shots at passing it. Rounds pay while the fix rate exceeds the newly-broken rate. Follow-up: “how would you know it helps?” → track first-pass acceptance and escape rate together, and ablate rounds from 3 to 1.

“How do you keep the review process sane?” Strong answer: cap diffs in flight per reviewer, read the diff stat before the diff, restrict blast radius, require independent tests, and treat the model’s written explanation as untrusted data. Watch review latency as the canary. Common wrong answer: “reviewers should be more careful.” That makes the constraint human and unmeasurable; shrink what must be verified instead.

Next: Multi-Head Attention from Scratch — the other half of the interview: writing the component yourself, from memory, with the shape and masking details right.