NL2SQL (Text-to-SQL)

TL;DR — A text-to-SQL system turns a question into a query against a real schema. The generation is the easy part; the work is grounding (schema linking), verification (does the query mean what the question meant), and safety (making it impossible for the model to write). The metric is execution accuracy, not string match, because two different queries can be equally correct — and it still cannot see a query that is valid, fast, and answers a question nobody asked. A wrong query is usually not an error: it is a plausible number. Reported error studies put schema errors at 81.2% of execution failures, and roughly half of all generated queries on BIRD containing at least one true error.

1. Why this exists

Someone asks the warehouse, in English: “How many UK customers churned last quarter?” The pipeline returns 412. There is no exception, no warning, and the number looks exactly like the right answer. It is wrong, because the model used orders.status = 'cancelled' as the definition of churn while the finance team means “no order in 90 days”, and because the join silently dropped customers with no orders at all.

That is what makes text-to-SQL harder than generation. The system’s failure mode is not a crash. It is a number.

Three distinct problems are stacked in that example, and they need different fixes:

• Grounding. The model had to decide that “churned” maps to a column or a query shape. That is schema linking plus a definitional question the database cannot answer by itself.
• Verification. The generated SQL parsed, ran, and returned rows. Nothing in the pipeline could distinguish “correct” from “well-formed”, because the specification was a sentence, not a test.
• Safety. The system had write access to a production warehouse. The same pipeline that produced SELECT count(*) could have produced DELETE — and the failure studies in §4 show that a meaningful share of generated queries do not even parse, let alone stop at reading.

So the design goal is not “generate SQL”. It is: make the space of executable queries smaller than the space of writable ones, and make the failure mode visible.

2. How it works

2.1 The pipeline

flowchart TD
    Q["question"] --> SL["schema linking:<br/>pick tables, columns, values"]
    SL --> G["generate SQL<br/>(LLM, few-shot or fine-tuned)"]
    G --> P["parse + compile check<br/>AST / EXPLAIN"]
    P -->|fail| R["repair with the error text"]
    P -->|ok| SAFE["safety gate:<br/>one statement, read-only role,<br/>row cap, statement timeout"]
    SAFE --> EX["execute on a read-only connection"]
    EX --> CHK["result sanity:<br/>row count, NULLs, magnitude, empty set"]
    CHK -->|suspicious| R
    CHK -->|plausible| OUT["table + natural-language explanation"]
    R --> G
    OUT --> CITE["show the SQL next to the answer"]

Two edges carry the value. The feedback edge from CHK back to G is the only mechanism that can catch a wrong-but-valid query, and its measured track record is mixed (§2.4). The CITE edge — showing the SQL — is not a UI nicety: it is the cheapest human verification, and it converts an unfalsifiable number into a reviewable artifact.

2.2 Execution accuracy, and why string match is not it

The standard metric is execution accuracy: run the predicted query and the gold query against the same database and compare the result sets.

$$\mathrm{EX}=\frac{1}{N}\sum _{i=1}^{N}1\left[\mathrm{exec}(q_i,D_i)=\mathrm{exec}(q_i^{\ast },D_i)\right]$$

• $N$ — the number of questions in the evaluation set.
• $q_i$ — the model’s query for question $i$; $q_i^{\ast }$ — the gold query.
• $\mathrm{exec}(\cdot ,D_i)$ — the multiset of rows the query returns against database $D_i$. Comparing results rather than text is the whole point: a CTE and a subquery that select the same rows score identically.
• $1[\cdot ]$ — the indicator, 1 when the two results match. The sum is therefore a count of matches, and dividing by $N$ makes it a rate in $[0,1]$.

The metric you must not use is exact set match on the SQL string. Zhong, Yu and Klein measured Spider’s clause-level string comparison against human-verified semantic equivalence and found false negatives at 2.5% on average and 8.1% worst case — correct queries scored wrong purely for being written differently.

But EX has the opposite bug, and it is the dangerous one. One database state is a sample of a query’s meaning, not its meaning:

• A query with a wrong WHERE clause can return the right rows if no row currently violates it. That is a false positive, and it inflates every published EX number to an unknown degree.
• A query using ORDER BY ... LIMIT 1 can be marked correct when ties make the top row arbitrary. The BIRD team patched exactly this after it was reported.

The principled fix is Zhong et al.’s test-suite accuracy: distil a small suite of synthetic databases with high code coverage for the gold query, and require the prediction to match the gold denotation on all of them. A query that is right by accident fails; a genuinely equivalent one passes. It costs more to build and it is the only version of the metric that luck cannot game.

What you report in practice. Say EX and test-suite accuracy together, and say which split. Spider 1.0’s leaderboard is frozen, so any number you quote is a historical snapshot, not a live ranking.

2.3 Schema linking is a recall problem with a hard ceiling

Schema linking is the step that decides which tables, columns, and literal values the generator even sees. Define, per question, the set of schema elements the gold query actually touches, $C^{\ast }$, and the set the linker selected, $C$:

$$\mathrm{link}\text{-}\mathrm{recall}=\frac{|C\cap C^{\ast }|}{|C^{\ast }|},\mathrm{compression}=\frac{|C|}{|S|}$$

• $C^{\ast }$ — the required columns and tables: everything the correct query reads. Ground truth you can compute offline from the gold SQL.
• $C$ — what the linker passed to the model. $S$ is the full schema, so the second quantity is the fraction of the schema you kept.
• $\mathrm{link}\text{-}\mathrm{recall}$ — the fraction of required elements the model can see. This is the term that matters, and its failure mode is not graceful: if a required column is missing from $C$, no amount of model quality recovers it, because the model is being asked to write about a column it has never been told exists. Recall sets a hard ceiling on EX. Precision of $C$ only costs tokens and confusion; recall costs the answer.
• $\mathrm{compression}$ — the token saving that motivated linking in the first place. The two are in direct tension: raising $|C|$ raises recall monotonically and lowers compression monotonically, so the operating point is chosen by the schema size, not by taste.

For a schema that fits in context, the tension can be removed entirely. Maamari et al., The Death of Schema Linking? found that current models use relevant schema elements well even surrounded by many irrelevant ones, so their pipeline forgoes schema linking whenever the schema fits the context window and relies on augmentation, selection, and correction instead; it ranked first on BIRD at the time of publication with 71.83% accuracy. The lesson is not “linking is dead” — it is that linking is a context-budget decision, not a correctness step. Link when the schema does not fit; do not link when it does, because every filtered-out column is a possible unrecoverable error.

The arithmetic that decides it is the schema’s token cost. For a schema with $T$ tables and $c_t$ columns in table $t$, a compact textual rendering costs roughly

$$\text{tokens}\approx \sum _{t=1}^T\left(\alpha +\beta c_t\right)$$

where $\alpha$ is the per-table overhead (the CREATE TABLE line, the comment) and $\beta$ the per-column cost (name, type, and any description or sample values). With $\alpha \approx 20$ and $\beta \approx 6$ (example, measured on a schema-rendering prompt), a 50-table, 1,000-column warehouse costs about $50\times 20+1000\times 6=7,000$ tokens, and an enterprise schema of 3,000 columns in 200 tables costs about 22,000 — which is where linking starts paying, and where Spider 2.0 says the problem actually lives.

2.4 The repair loop: an optimistic model and its measured correction

The appealing argument for execution feedback is that the database is a verifier. If a query fails to parse, the error message is ground truth, so you feed it back and resample. Model it: if each attempt independently succeeds with probability $p$, then $k$ attempts give

$$P(\text{at least one correct})=1-(1-p{)}^k$$

• $p$ — the single-attempt success rate, the thing that looks like your headline number.
• $k$ — how many attempts, or repair rounds, you are willing to pay for.
• The independence assumption is doing all the work, and it is false for self-repair: the repair prompt contains the model’s own wrong query, so the failures are correlated. Two attempts that share a wrong schema-linking decision fail together, which is precisely why observed gains from $k$-round self-correction fall far short of this curve.

That is not a hand-wave; it is measured. Shen et al., Understanding, Detecting, and Repairing Real-World In-Context-Learning-Based Text-to-SQL Errors (FSE 2026) hand-labelled 4,602 incorrect queries across four ICL techniques, two benchmarks, and two model settings — about 840 person-hours — and then measured five published repair methods against that taxonomy. The numbers:

• Repairs fixed only 10.9-23.3% of the erroneous queries (75-161 out of 690 in their main setting).
• They introduced 5.3-40.1% as many new errors as they fixed (4-55).
• They cost 1.03-3.82x the latency (1.38-5.14 s).

Read as a precision, the repair step is $F/(F+B)$ with $B/F$ between 0.053 and 0.401 — so 71% to 95% of its changes are improvements. It is net-positive and not free, and the honest test is the marginal cost:

$$\text{keep repairing}⟺{\Delta }_{\text{correct}}\cdot V>k\cdot C_{\text{round}}$$

where ${\Delta }_{\text{correct}}$ is the queries you actually fix, $V$ the value of a correct answer, and $C_{\text{round}}$ the latency-plus-tokens cost of one round. “4% more execution accuracy” says nothing until $V$ and $C_{\text{round}}$ appear in the same sentence.

2.5 What actually stops a write

Read-only enforcement is a property of the connection and the role, never of the prompt. Three layers, in increasing order of how much they depend on you being right:

1. A database role without write grants. GRANT SELECT on the specific schemas, nothing else. This one survives every bug in your application, including the one where you call the wrong connection.
2. A read-only transaction or session. SET SESSION CHARACTERISTICS AS TRANSACTION READ ONLY, or ALTER ROLE ... SET default_transaction_read_only = on, rejects INSERT, UPDATE, DELETE, MERGE, COPY FROM, every CREATE/ALTER/DROP, TRUNCATE, GRANT/REVOKE, and EXPLAIN ANALYZE of those. The PostgreSQL manual is explicit that this is “a high-level notion of read-only that does not prevent all writes to disk” — sequence advancement and temporary tables are the classic escapes — which is why it is layer 2.
3. A statement timeout and a row cap. statement_timeout aborts a statement that runs too long; it is measured from arrival at the server, and zero (the default) disables it, so an unconfigured system has no ceiling at all.

Two things that look like safety measures and are not:

• A LIMIT clause is not a cost cap. SELECT * FROM events LIMIT 10 over an unindexed billion-row table still reads until it finds ten matching rows. Bound the work with a timeout and an EXPLAIN-based plan check (reject a sequential scan over a large table unless the question says “all”).
• A regex that rejects the word DELETE. Trivially bypassed by a dialect-specific spelling, a comment, or a hyphenated keyword — and it makes a team believe it has a control it does not have.

What does help before execution is a compile: parse and plan the statement without running it. EXPLAIN <sql> does this in SQLite and in PostgreSQL (inside a read-only transaction). It catches syntax errors, unknown columns and type mismatches — the 81.2% of failures that are schema errors — for the cost of a planning pass.

3. Implementation

The listing is a complete guard-and-execute harness: Python 3.11+ with the standard library only, using sqlite3. It is runnable as written, and the numbers in §4 come from running it.

"""Text-to-SQL safety gate + execution-accuracy harness. Python 3.11+, stdlib."""
import sqlite3
import time

DDL = """
CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT, country TEXT);
CREATE TABLE orders (id INTEGER PRIMARY KEY, customer_id INTEGER, total_cents INTEGER,
                     status TEXT, created_at TEXT);
INSERT INTO customers VALUES (1,'Ada','UK'),(2,'Grace','US'),(3,'Alan','UK');
INSERT INTO orders VALUES (1,1,1200,'shipped','2024-01-05'),
                          (2,1,999,'pending','2024-02-11'),
                          (3,2,4500,'shipped','2024-03-02'),
                          (4,3,700,'cancelled','2024-03-19');
"""


class QueryRejected(Exception):
    """Raised by the gate before any SQL reaches the database."""


def build(path=":memory:"):
    conn = sqlite3.connect(path)
    conn.executescript(DDL)
    return conn


def read_only(path):
    """A connection that cannot write even if the gate is bypassed."""
    conn = sqlite3.connect(f"file:{path}?mode=ro", uri=True)   # layer 1
    conn.execute("PRAGMA query_only = ON")                     # layer 2
    return conn


def gate(sql, row_cap=500):
    """Return the SQL to execute, or raise QueryRejected.

    Three jobs, in order of how much damage they prevent:
      1. one statement only, so 'SELECT 1; DROP TABLE orders' cannot get in;
      2. wrapping the body in a subquery, which forces the inner statement to be
         a SELECT -- DROP/DELETE/UPDATE do not compile inside one;
      3. a row cap, so the result cannot be unbounded.
    The gate is defence in depth, not the only defence: the connection is
    read-only, so a statement that slips past still cannot write.
    """
    body = sql.strip().rstrip(";")
    if ";" in body:
        raise QueryRejected("multiple statements")
    if not body:
        raise QueryRejected("empty statement")
    return f"SELECT * FROM ({body}) LIMIT {row_cap}"


def run(sql, conn, timeout_s=2.0, row_cap=500):
    """Compile, bound, and execute with a wall-clock abort."""
    try:
        conn.execute("EXPLAIN " + gate(sql, row_cap))    # compile only, no rows
    except sqlite3.Error as exc:
        raise QueryRejected(f"does not compile: {exc}") from exc
    deadline = time.monotonic() + timeout_s
    conn.set_progress_handler(lambda: 1 if time.monotonic() > deadline else 0, 1000)
    try:
        return conn.execute(gate(sql, row_cap)).fetchall()
    except sqlite3.OperationalError as exc:
        raise QueryRejected(f"execution aborted: {exc}") from exc
    finally:
        conn.set_progress_handler(None, 0)               # always clear the hook


def execution_accuracy(pred, gold):
    """Spider's comparison: the two result multisets must match."""
    return sorted(map(tuple, pred)) == sorted(map(tuple, gold))

Running it against the two-table fixture:

conn = build(path); conn.commit(); conn.close()
ro = read_only(path)
run("SELECT country, COUNT(*) AS n FROM customers GROUP BY country", ro)   # works
run("DROP TABLE orders", ro)                       # QueryRejected: does not compile
run("SELECT 1; DROP TABLE orders", ro)             # QueryRejected: multiple statements
run("SELECT salary FROM customers", ro)            # QueryRejected: no such column
run("SELECT * FROM orders", ro, row_cap=2)         # 2 rows, not 4
run("WITH RECURSIVE c(x) AS (SELECT 1 UNION ALL SELECT x+1 FROM c) "
    "SELECT count(*) FROM c", ro, timeout_s=0.25)  # QueryRejected: interrupted

The printed outcomes are these, and each one is a distinct control:

And with the gate deliberately bypassed, the connection still refuses:

ro.execute("DELETE FROM orders")   # sqlite3.OperationalError: attempt to write
                                   # a readonly database

The knobs that matter

Two notes on portability before you copy the gate. The subquery wrapper is a SQLite trick; PostgreSQL also accepts SELECT * FROM (<query>) AS q LIMIT n but requires the alias and rejects a subquery with duplicate output column names, so test it against your real dialect. And the row cap must be an integer you produced, never a string interpolated from user input.

4. Numbers and trade-offs

The benchmarks

The last row is the one to internalise. The 86.6 → 10.1 drop is not a model regression; it is the difference between a benchmark and a job, and it is why a “we’ll just use a frontier model” plan is not a plan.

Where generated queries actually go wrong

From the FSE 2026 error study (arXiv:2501.09310), over 6,136 generated queries on BIRD with reliable correctness judgement:

• 47.8% contained at least one true error on BIRD; 26.8% on Spider. BIRD is harder because its databases are large, dirty, and require external knowledge.
• Of execution failures, 81.2% are schema errors and 18.8% are syntax errors. More than 40% of the syntax errors in BIRD are function hallucination — calling YEAR(x) on SQLite, where it does not exist.
• 26.0% of all errors (25.8% on Spider) are intent-independent: detectable without ever reading the question. That is the category your gate can catch for free.
• The authors also report that around 15.4% of queries flagged as “not an error” by the existing judges were in fact fine — the benchmark’s own correctness labels are noisy, which is worth remembering before you tune to the third decimal.

Cost, derived

The dominant cost is prompt tokens, and the schema is the biggest term. The 50-table / 1,000-column rendering in §2.3 is ~7,000 schema tokens per request: at 10,000 requests/day, 70M schema tokens/day before the question is asked. Schema retrieval or prefix caching attacks that directly; a repair round adds a second full pass carrying the failed query and the error. This is why schema selection reports a 5x token reduction (CHESS, arXiv:2405.16755) alongside a ~2-point accuracy gain — and why that trade exists only when the schema does not fit.

Execution is milliseconds and the model call hundreds to thousands, so resilience work belongs in the verification path, not in retry loops.

5. Pitfalls

Confident wrong SQL. Symptom: a plausible number with no error anywhere, which a human only catches by reading the SQL or knowing the domain. This is the default failure, not an edge case — roughly half of generated queries on BIRD contain a true error. Fix: always render the SQL next to the answer, and run result sanity checks (empty result set, row count far outside historical range, a NULL-inflated aggregate) that route to a human rather than to a user.

Schema drift. Symptom: a previously working pipeline starts failing compilation on a column that was renamed, or worse, silently returns wrong results because a column’s meaning changed while its name did not. Fix: the schema prompt is generated from the live catalogue, never hard-coded; and the eval set is versioned with the schema, so a drift shows up as a metric drop rather than as a support ticket.

A destructive query getting through. Symptom: data is gone. The classic path is not a malicious prompt but a well-meaning DELETE in a repair round, an INSERT into a temp table, or a SELECT ... INTO in a dialect that supports it. Fix: three layers from §2.5, and the first one is a role without write grants, because it is the only one that does not depend on your gate being correct.

Treating LIMIT as a cost control. Symptom: the warehouse alert fires on a query your system generated with LIMIT 10 in it. LIMIT bounds the output, not the work. Fix: statement timeouts plus a plan check.

Trusting the repair loop’s headline. Symptom: the loop’s “queries fixed” number is reported without the “queries newly broken” number, and the system looks like it improves more than it does. The measured ratio is 5.3-40.1 new breakages per 100 fixes. Fix: report both, and gate the loop on the marginal cost inequality in §2.4.

Schema linking that filters out the answer. Symptom: the same question fails repeatedly across prompt variants, and the correct column appears nowhere in the prompt. This is an unrecoverable failure: no model improvement helps. Fix: log $\mathrm{link}\text{-}\mathrm{recall}$ against gold SQL on your eval set, and treat a drop in it as a hard regression, separate from EX.

Ambiguous definitions treated as a modelling problem. Symptom: the system is “wrong” on churn, revenue and active user, but every query faithfully reads the question. Fix: this is a metric-definition problem, not a generation one. Put named metrics in a semantic layer and have the model select one rather than compose its formula.

6. Using it in a project

Reach for NL2SQL when the questions are answerable by a single query against a schema you control, the schema is stable enough to describe, there is a way to check results (gold answers, a semantic layer, or a human in the loop), and the volume of ad-hoc questions justifies the build.

Do not reach for it when the question needs an analyst. “Why did revenue drop last month?” is not a query; it is a hypothesis, several queries, and a judgement about which of them is relevant. Systems that answer it by free-styling SQL produce confident narratives with no falsifiable claim in them. Route those to a dashboard, a notebook, or a person.

Four more cases where SQL generation is the wrong tool:

• The question is repetitive. If 80% of traffic is ten questions, those are ten saved queries or parameters in a semantic layer. A cached template is free, exact, and reviewable; a model call is none of those. This is rung 1 on the decision ladder.
• The schema is small and the users are known. A query builder or a set of filters is faster, safer, and has a discoverable interface. Text-to-SQL is for when the interface cannot be enumerated.
• There is no read replica. If the generation path can reach the primary, the safety work in §2.5 is not optional and it is not cheap; do not start until the read-only role exists.
• The metric definitions are contested. Build the semantic layer first. A model that composes revenue from three columns will be wrong in a different way each week, and the argument will be with the model instead of with the definition.

Build in this order: the read-only role and the eval set (150-300 hand-labelled questions, schema version pinned); then a zero-shot full-schema baseline; then schema selection, only if the schema does not fit; then the result-sanity checks and the SQL-in-the-answer affordance; then multi-candidate generation; and only then a repair loop, with both of its numbers tracked.

7. How you would prove it worked

Primary metric: execution accuracy on a versioned, hand-labelled set, split by difficulty band (the BIRD taxonomy of simple / moderate / challenging is a good default). Report test-suite accuracy on the same set if you can afford it, because EX alone cannot distinguish “right” from “lucky”.

Secondary metrics, which move independently:

• Valid-SQL rate — the cheapest number to move and the least correlated with correctness. A dashboard guardrail, not a goal.
• Link recall against gold SQL, per question: a drop predicts an EX drop.
• Gate block rate and rejection reasons — a spike in multiple statements is a canary for a prompt regression.
• p50/p95 latency and schema tokens per request — the two costs the accuracy work trades against.

Baseline: the zero-shot, full-schema prompt, same model, same pinned eval set. Linking, candidates and repair each earn their place against it and its cost, one axis at a time.

Ablations, in priority order: (1) schema linking on/off at fixed model, which tests §2.3 directly; (2) the repair loop on/off, reporting both fixed and newly-broken; (3) $k\in \{1,3,5\}$ candidates with self-consistency, to find where returns stop; (4) EXPLAIN plan rejection on/off, measuring warehouse load avoided against legitimate queries rejected.

Reporting template (example — replace every number). “On a 240-question labelled set against the production schema (sales_v14), execution accuracy went from 61.5% (zero-shot, full schema) to 74.2% with schema selection, and to 76.7% after adding two-candidate voting. Test-suite accuracy on the same set was 71.9%, so roughly 5 points of the headline EX were luck on a single database state. Valid-SQL rate is 99.1%. The repair loop fixed 34 queries and broke 9 (repair precision 0.79) for +1.4 s p95, and we kept it. Link recall is 0.98; the residual failures are ‘active user’ and ‘churn’, which is a definition problem and not a generation problem — the next fix is a semantic layer, not a bigger model.”

That report is defensible because every number has a baseline next to it, the metric with the flattering story (EX) is paired with the one that can contradict it, and the residual failures are attributed to the right layer.

8. Interview drill

“Which metric for text-to-SQL?” Strong answer: execution accuracy, because two different queries can be equally correct and string match punishes them — 2.5% false negatives on average, 8.1% worst case, against Spider’s own metric. Then name EX’s weakness: one database state samples the query’s meaning, so it admits false positives, and test-suite accuracy is the fix. Common wrong answer: “exact match on the SQL.” It measures syntax agreement, and penalises every rewriting-based method for being right in a different shape.

“Where does generated SQL go wrong?” Strong answer: name the taxonomy and the split. On BIRD, 81.2% of execution failures are schema errors (non-existent or mismatched columns, unused aliases, ambiguous references) and 18.8% are syntax, with function hallucination over 40% of the syntax errors. Separately, 26.0% of all errors are intent-independent — findable without the question — and roughly half of BIRD queries contain at least one true error. Common wrong answer: “hallucinated tables.” True but narrow; it is one subcategory of one category, and it misses the large intent-independent share that a deterministic gate can catch for free.

“How do you make it safe to run against production?” Strong answer: three layers, in order of reliability: a role with SELECT and nothing else; a read-only transaction, noting that PostgreSQL documents it as not preventing all writes to disk; and a statement timeout plus a plan check, because LIMIT bounds output rather than work. Add a compile-only EXPLAIN first. Common wrong answer: “we regex the SQL for DELETE and DROP.” That fails silently and creates false confidence; the grant is the control.

“The schema is 3,000 columns. What do you do?” Strong answer: link, because it no longer fits — and treat it as a token-budget decision, not a correctness step. Measure link recall against gold SQL: a missing required column is an unrecoverable ceiling on accuracy, while extra columns only cost tokens. Follow-up: “and if it fits in context?” → do not link. Models use relevant elements well amid irrelevant ones, and every filtered column is a new way to fail.

“You add execution feedback as a repair loop. How much does it help?” Strong answer: it is net-positive but modest, and you must report both directions: measured repairs fix 10.9-23.3% of erroneous queries and introduce 5.3-40.1% as many new errors, at 1.03-3.82x latency. Then explain why the naive $1-(1-p{)}^k$ model overstates it: the repair prompt contains the model’s own wrong query, so failures across rounds are correlated. Common wrong answer: “the database is a verifier, so we just retry until it passes.” The database verifies executability, not intent; a wrong query that runs passes every retry.

“How do you evaluate correctness when the database is dirty?” Strong answer: run the prediction against multiple synthetic database states so a right-by-accident query fails; and accept that labels are noisy — the error study found ~15.4% of queries flagged as errors were not errors — so hand-audit a sample and version the eval set with the schema. Trap: a single EX number on a live production database. It is unreproducible and the ground truth moves under you.

Next: AI-Assisted Coding — the same three problems in a different medium: grounding on a repository, verifying an edit without a gold query, and keeping the model away from the parts of the system that cannot be rolled back.