Guardrails

TL;DR — An LLM cannot reliably police itself. Models make no architectural distinction between instructions and data, so any text that reaches the context window is a potential instruction. Researched defences, including human red-teaming, have been bypassed at above 90% success by adaptive attackers. The correct posture is therefore not “filter harder” but assume the model will be fooled, and make being fooled non-catastrophic. Three layers — input, decision, output — plus the lethal trifecta / Agents Rule of Two constraint, which you check by inspecting your tool list rather than the model’s behaviour. The strongest controls remove capability; they do not add checks.

1. Why this exists

Four incidents, four delivery surfaces, one ending.

• Amazon Q, July 2025 — an attacker committed a destructive system prompt to the VS Code extension repo; it failed only on a syntax error. Separately, a runtime injection made the agent execute arbitrary code.
• Microsoft 365 Copilot, 2025 (“EchoLeak”) — a crafted email triggered zero-click exfiltration, bypassing both the injection classifier and the link-redaction filter, with no user interaction.
• A production database, 2025 — Cursor, wired to a Supabase MCP server running with the service_role key, dumped the database, bypassing row-level security entirely because the credential bypassed it.
• postmark-mcp, 2025 — a malicious npm package BCC’d email to an attacker across an estimated 300 organisations, doing exactly what it advertised plus one thing it did not.

None is “the model was too weak”. Each is a system failure: the model had a capability, an attacker reached it with text, and nothing between the model and the consequence was standing.

Why it is structural, not a bug

“LLMs follow instructions in content… they don’t just follow our instructions. They will happily follow any instructions that make it to the model, whether or not they came from their operator… LLMs are unable to reliably distinguish the importance of instructions based on where they came from.” — Willison

OWASP states it technically: “LLMs make no architectural distinction between ‘instructions’ and ‘data’ (both are tokens on the same stream), so there is no clean equivalent to parameterized queries” (LLM01:2026, [as of 2026-10]).

Compare SQL injection, solved with parameterised queries: a structural separation between code and data that the interpreter enforces. There is none for a transformer. <system> tags, role hierarchies, and “ignore previous instructions” are prompt-level conventions, not enforced boundaries. That difference is the chapter.

Where this matters for you. The interview question is never “how do guardrails work”. It is “an attacker can put text in a page your agent reads. What stops them?” If your answer is a filter, you have failed the question.

2. How it works

2.1 Direct versus indirect injection

• Direct (“jailbreaking”) — the user tries to override the system prompt. Attacker and principal are the same party.
• Indirect — a third party plants text in data the agent later retrieves: a web page, an email, an issue title, a tool result, a retrieved chunk, an image. The user is a victim. OWASP: “The user did not supply or see those instructions.”

Willison’s reason for insisting on the distinction: “Developers who misunderstand these terms and assume prompt injection is the same as jailbreaking will frequently ignore this issue as irrelevant to them.” Direct injection violates a trust boundary held by a principal; indirect injection crosses a data channel. Only the latter makes a tool-using agent dangerous.

OWASP decomposes a payload along three axes, which is a good threat-model template:

• Delivery surface — direct input, retrieved content, tool output, tool connection channel, or persistent memory.
• Propagation — single-shot, multi-step kill chain, cross-session through memory or a RAG corpus, or self-replicating across agents.
• Encoding — plain text, base64, invisible Unicode, multimodal, or a low-resource language. This axis is why text filters are insufficient: OWASP documents smuggling via tag-block (U+E0000-U+E007F), variation-selector (U+FE00-U+FE0F), and zero-width characters (U+200B, U+200C, U+200D, U+2060), and the 2024 M365 Copilot ASCII-smuggling proof of concept used them to exfiltrate a Slack MFA code.

2.2 Why an LLM cannot reliably police itself

(a) The architectural argument. Quoted in section 1: no provenance channel. Instruction-hierarchy tricks are conventions the model usually honours, which is worse than useless, because it manufactures the belief that a boundary exists.

(b) The empirical argument. Nasr, Carlini et al., “The Attacker Moves Second” (arXiv:2510.09023, 2025-10-10), with authors from OpenAI, Anthropic, and Google DeepMind:

“By systematically tuning and scaling general optimization techniques—gradient descent, reinforcement learning, random search, and human-guided exploration—we bypass 12 recent defenses … with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates.”

The per-defence jumps are the part to internalise: Spotlighting 28% → 99%; RPO 0% → 99%; Circuit Breaker 8% → 100%; StruQ 62% → 100%; PIGuard 0% → 71%; Data Sentinel 0% → 80%; and human red-teaming 0% → 100%. The conclusion drawn by whoever charted those results is right: “static example attacks — single string prompts designed to bypass systems — are an almost useless way to evaluate these defenses.”

(c) The vendor-claim argument. Guardrail products “almost always carry confident claims that they capture ‘95% of attacks’ or similar… but in web application security 95% is very much a failing grade.” Section 2.5 puts a number on why.

The engineering conclusion is not “filter harder.” It is: assume the model will be fooled, and make being fooled non-catastrophic.

2.3 The three layers

flowchart LR
    IN["Untrusted input"] --> L1["1. Input<br/>labelling, Unicode<br/>normalisation, classifier"]
    L1 --> M["Model proposes<br/>an action"]
    M --> L2["2. Decision<br/>policy engine, least privilege,<br/>approval gate, sandbox"]
    L2 --> X["Side effect"]
    M --> L3["3. Output<br/>schema validation, encoding,<br/>path/query canonicalisation"]
    L3 --> DOWN["Downstream system"]

Ranked by how much each layer actually buys you, given section 2.2:

The ordering is counter-intuitive and it is the point: everyone builds the input layer first, and it has the worst cost/benefit.

2.4 The lethal trifecta and the Agents Rule of Two

Both are design constraints, not detection mechanisms. You check them by inspecting your tool list, not your model’s behaviour — the only kind of safety property that survives section 2.2.

Lethal trifecta (Willison, 2025-06-16 — his framing, not Anthropic’s): access to your private data, exposure to untrusted content, and the ability to communicate externally in a way that could steal it. “If your agent combines these three features, an attacker can easily trick it into accessing your private data and sending it to that attacker.” He states the scope limit himself: it “only covers the risk of data exfiltration.”

Agents Rule of Two (Meta AI, 2025-10-31) fixes that by adding state change. Agents “must satisfy no more than two” of: [A] can process untrustworthy inputs; [B] can access sensitive systems or private data; [C] can change state or communicate externally. “If an agent requires all three without starting a new session … then the agent should not be permitted to operate autonomously and at a minimum requires supervision.” Willison endorses it, then dissents from one cell — input plus state change is marked safe, “but that’s not right”; Meta later relabelled those intersections “lower risk”.

OWASP’s 2026 entry adopts it as a floor: an agent with all three “has the conditions for high-impact exploitation, and removing any one leg removes them.” In practice: mark every tool A, B, or C; any session reaching all three needs a human gate on the irreversible [C] action.

2.5 The risk arithmetic

Most guardrail discussions fail here, treating a classifier as binary and a stack of classifiers as additive. Neither is true.

One layer, many exposures. Let $d$ be the probability that a given malicious payload fails to get through a control — its detection rate — and $N$ the number of independent untrusted items the agent processes in a task. If one bypass compromises the task:

$$P_{\text{bypass}}(N)=1-d^N$$

• $d$ — the control’s detection rate against the attacks it is being tested with. It is a property of the adversary, not a constant of the system, which is exactly what section 2.2 established.
• $N$ — the number of independent exposures. This is the term that turns a “good enough” classifier into a failing one, and the term you can actually reduce: fetching one page instead of twenty cuts the exponent.
• $d^N$ — the probability that every item is caught, which assumes independence. Real documents from one source are correlated, so the true $P_{\text{bypass}}$ is higher; treat this as a lower bound on risk, not a prediction.
• $1-(\cdot )$ — “at least one got through”. An attacker needs one.

(example) At $d=0.95$ — well above what the adaptive literature supports — and $N=20$ untrusted documents per task, $P_{\text{bypass}}=1-0.95^{20}=0.64$. Two tasks in three see a bypass. At $N=5$ the same classifier gives 0.23. This is the arithmetic behind “95% is a failing grade”: the classifier is not the problem, the exponent is.

Many layers. If layers were independent, a stack of $L$ would give $P_{\text{bypass}}={\prod }_{j=1}^L(1-d_j)$. They are not: the payload that evades one classifier usually evades a second trained on the same data, and an attacker who reads your design defeats both. The honest model is one parameter, $P_{\text{bypass}}=p_{\text{pass}}$, the probability that a specifically crafted payload clears the whole stack — above 0.9 for most researched defences. Do not multiply; measure the stack as a unit under adaptive attack (section 7).

Risk is probability times impact. Once you accept you cannot drive $p_{\text{pass}}$ to zero, ask what it multiplies:

$$R=p_{\text{pass}}\times I$$

• $p_{\text{pass}}$ — as above. Input-layer controls move this, weakly.
• $I$ — the impact of one successful injection: rows readable, dollars transferable, systems alterable. Decision-layer controls move this, decisively.
• $R$ — expected loss per task. Reducing $I$ by 100x beats reducing $p_{\text{pass}}$ by 2x, and it is far easier, because $I$ is a property of your tool list rather than of an adversarial arms race.

(example) A tool that can SELECT * from production has $I$ of “everything”; the same capability scoped to one tenant and one table has $I$ of “that tenant’s data”. Both are reached by the same $p_{\text{pass}}$.

The approval gate, costed. If a fraction $g$ of actions are gated and each review takes $t_r$ seconds:

$$\Delta L=g\cdot n_a\cdot t_r$$

• $g$ — the fraction of tool calls that are irreversible, external, or privileged. The knob: gating everything drives $g\to 1$ and makes the agent unusable.
• $n_a$ — actions per task. 50 actions at $g=0.2$ is 10 interruptions.
• $t_r$ — review time per decision, floored around 10-30 seconds for a reviewer who actually reads the arguments.
• $\Delta L$ — added latency per task. (example) $g=0.2$, $n_a=50$, $t_r=20$s gives 200s of pure review.

Two consequences. Gate by consequence, not category: reversible actions (a draft, a store-credit refund) auto-approve; irreversible ones (an external payout, a production delete) route to a human — OWASP’s “graduated enforcement (audit, warn, block, escalate)”. And approval fatigue is a vulnerability: human red-teaming went 0% to 100% bypass under adaptive attack, and a reviewer shown 200 prompts is not a control.

2.6 Least-privilege tool design

OWASP LLM03:2026 names three root causes, each with a different fix:

Permissions is the strongest, being enforced by a system that cannot be argued with. The entry’s fifth mitigation is the one people miss: “Execute tools in the user’s context” — track “user authorization and security scope to ensure actions taken on behalf of a user are executed on downstream systems in the context of that specific user.” The Cursor/Supabase incident is exactly this: a service_role key makes every query ambiently privileged, so row-level security never applies. Two techniques follow: poka-yoke the interface, changing arguments so a mistake is hard to make (Anthropic’s SWE-bench example, requiring absolute file paths, “eliminated a whole class of model errors”, because a tool that cannot express the dangerous input cannot receive it), and complete mediation — “Implement authorization in logic rather than relying on an LLM to decide if an action is allowed or not.”

The anti-pattern, plainly: fewer, narrower tools beat one powerful tool. Anthropic’s named failure mode is “bloated tool sets”: “If a human engineer can’t definitively say which tool should be used in a given situation, an AI agent can’t be expected to do better.”

2.7 Sandboxing

A sandbox’s job is to make the [B] leg unreachable and the [C] leg auditable, so the trifecta cannot complete. The mechanisms differ in what kind of boundary they provide, and that is the only distinction that matters.

The engineering rule. Sandbox all model-generated code, with a boundary at least as strong as the trust you place in the model’s output — which, per section 2.2, is zero. And the rule that catches most deployments: a sandbox with network egress and a shared filesystem containing secrets is not a sandbox, it is a seatbelt. Default to no network, an ephemeral filesystem, no host mounts, and a credential injected per invocation rather than per agent. The trifecta tells you which two legs to cut.

2.8 Patterns that defeat injection structurally

Beurer-Kellner et al., “Design Patterns for Securing LLM Agents against Prompt Injections” (arXiv:2506.08837), states the guiding principle:

“once an LLM agent has ingested untrusted input, it must be constrained so that it is impossible for that input to trigger any consequential actions.”

It is honest about the ceiling: “we believe it is unlikely that general-purpose agents can provide meaningful and reliable safety guarantees” for this class of model.

1. Action-Selector — the agent triggers tools but never sees their responses: an “LLM-modulated switch statement”, immune by construction because no channel runs from untrusted data back to action selection.
2. Plan-Then-Execute — output flows back but cannot influence which actions are chosen. Output can corrupt an email body but not its recipient. The best cost/benefit for most products.
3. LLM Map-Reduce — sub-agents consume untrusted content in isolation and return safely-aggregated results; the coordinator never touches the tainted text.
4. Dual LLM — a privileged LLM coordinates a quarantined one that returns symbolic variables (\$VAR1 for a summarised page) it can pass on “without being exposed to that tainted content itself.”
5. Code-Then-Execute — Google DeepMind’s CaMeL (arXiv:2503.18813): the privileged LLM generates code in a sandboxed DSL “designed to enable full data flow analysis, such that any tainted data can be marked as such and tracked.”
6. Context-Minimisation — drop content from context over time (reduce a request to a database query, then drop the prompt before returning results).

The trade, in the paper’s own terms: these patterns “offer a valuable trade-off between agent utility and security” by “limiting the ability of agents to perform arbitrary tasks.” Hence the one-line summary: security in agent design comes from removing capabilities, not from adding checks.

2.9 The frameworks to cite

OWASP Top 10 for LLM Applications 2026, published 2026-08-04, with a renumbering most material still gets wrong ([as of 2026-10]):

The agent-relevant entries: LLM01 (the input boundary), LLM03 (an agent’s tool call is a privileged action), LLM06 (step caps and budgets), LLM08 (tool inventories and internal URLs live in the system prompt, so design assuming they are discoverable, and never put credentials there), and LLM10 (a tool call is output that reaches a downstream system).

There is also the OWASP Top 10 for Agentic Applications: ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain, ASI05 Unexpected Code Execution, ASI06 Memory and Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human Agent Trust Exploitation, ASI10 Rogue Agents.

2.10 Unbounded consumption is a safety property

LLM06 is not merely a cost issue: an unbounded loop consumes until something else fails, and an attacker triggers it for free. Deterministic controls, none consulting the model: a step cap (ReAct’s backoff thresholds were 7 steps on HotpotQA and 5 on FEVER, “only … 0.84% and 1.33% respectively” of correct trajectories — long trajectories were almost never productive), a token budget per task checked by the harness, wall-clock and tool-call caps, a cost cap with a hard kill, and loop detection, since ReAct’s most common failure is exact repetition of a prior thought plus action.

3. Implementation

A three-layer validation stack. Dependency: pydantic 2.14.1 ([as of 2026-10]).

3.1 The input layer: provenance, not filtering

# guardrails_input.py — stdlib; no model calls
import re
import unicodedata

# Invisible carriers documented as smuggling channels (OWASP LLM01:2026, ex. 5).
_INVISIBLE = re.compile(
    "[\u200b-\u200f"           # zero-width space/joiners, LRM/RLM
    "\u2060-\u2064"            # word joiner, invisible operators
    "\ufe00-\ufe0f"            # variation selectors
    "\U000e0000-\U000e007f]"   # tag block; NOTE the capital \U and 8 digits
)


def normalise_untrusted(text: str) -> str:
    """Strip invisible carriers; normalise Unicode.

    This does NOT detect injection: a visible-text payload passes unchanged. It
    removes a *channel*, not a *class* of attack.
    """
    return unicodedata.normalize("NFKC", _INVISIBLE.sub("", text))


def as_untrusted_block(source: str, text: str) -> str:
    """Label external content so provenance is explicit.

    Delimiters reduce non-adaptive attack success; they do NOT create a boundary
    (StruQ was bypassed under adaptive attack). Friction, not enforcement. The
    fence comes from `source`, never from attacker-controlled text.
    """
    return (f'<untrusted_content source="{source}">\n'
            f'{normalise_untrusted(text)}\n</untrusted_content>')

Three design points. Strip, do not sanitise — deleting invisible characters removes a delivery channel at zero false-positive cost, since legitimate prose does not need them; detecting meaning is the layer that fails. Label, and never interpolate into a system position — a tool result must never be formatted as a system message, however convenient the API makes that. And the block is deliberately ugly: a visible fence is honest about its own weakness.

3.2 The decision layer: a deterministic policy engine

# guardrails_policy.py — pydantic 2.14.1
from enum import Enum
from typing import Any, Callable

from pydantic import BaseModel, Field


class Risk(str, Enum):
    READ = "read"                  # reversible, no external communication
    WRITE = "write"                # reversible, internal state
    IRREVERSIBLE = "irreversible"  # external, destructive, or value-moving


class Decision(str, Enum):
    ALLOW = "allow"
    REQUIRE_APPROVAL = "require_approval"
    DENY = "deny"


class ToolSpec(BaseModel):
    name: str
    risk: Risk
    # Only these principals may invoke it. Empty means "nobody": a forgotten tool
    # is denied, not allowed.
    allowed_principals: set[str] = Field(default_factory=set)
    # Deterministic checks over ATTACKER-CONTROLLED arguments: path containment,
    # amount caps, recipient allowlists. Each returns a problem string or None.
    validators: list[Callable[[dict[str, Any]], str | None]] = Field(
        default_factory=list, exclude=True
    )


def action_identity(tool: str, arguments: dict[str, Any]) -> str:
    """Stable hash of the tool plus canonicalised arguments. Serves three roles:
    binds an approval to one action, is the idempotency key, and is the audit-log
    primary key."""
    import hashlib
    import json
    blob = json.dumps({"tool": tool, "args": arguments},
                      sort_keys=True, default=str)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


def decide(spec, principal, arguments, approvals_granted):
    """Complete mediation: every privileged call passes through here. No model,
    no confidence score, no natural-language reasoning -- those are the things an
    attacker can influence."""
    if principal not in spec.allowed_principals:
        return Decision.DENY, f"principal {principal!r} may not call {spec.name}"
    for check in spec.validators:
        problem = check(arguments)
        if problem:
            return Decision.DENY, problem
    if spec.risk is Risk.READ:
        return Decision.ALLOW, "read-only"
    if spec.risk is Risk.WRITE:
        return Decision.ALLOW, "reversible and in-scope"
    # IRREVERSIBLE: gate on consequence, and bind the approval to THIS action.
    action_id = action_identity(spec.name, arguments)
    if action_id in approvals_granted:
        return Decision.ALLOW, "human approved"
    return Decision.REQUIRE_APPROVAL, f"approval required for {action_id}"

What each parameter is for:

• risk — a hand-assigned class, not a model judgement. It decides whether a human is in the loop, so it must be reviewable in a pull request.
• allowed_principals — caller identities permitted to invoke the tool, defaulting to empty: this is how “execute tools in the user’s context” becomes code rather than advice.
• validators — deterministic checks on the raw arguments, the only checks that survive an adaptive attacker because they never consult the model.
• approvals_granted — a set of action identities, not a boolean: approving “delete the staging bucket” must not approve a later deletion of a different bucket.
• Decision — three outcomes, with REQUIRE_APPROVAL the default for the irreversible class without an approval. A system that fails open here will eventually perform an unapproved deletion.

3.3 The output layer: model output is attacker-controlled

# guardrails_output.py — pydantic 2.14.1
import os
import re
from pathlib import Path

from pydantic import BaseModel, Field, field_validator


class FileEdit(BaseModel):
    """The schema a model's 'edit a file' call must satisfy."""

    path: str
    contents: str = Field(max_length=200_000)   # unbounded output = LLM06

    @field_validator("path")
    @classmethod
    def path_inside_workspace(cls, v: str) -> str:
        # Canonicalise BEFORE checking containment: `../` and symlinks defeat a
        # string-prefix check. Reject NUL early; the OS truncates there.
        if "\x00" in v:
            raise ValueError("NUL byte in path")
        root = Path(os.environ["WORKSPACE_ROOT"]).resolve()
        candidate = (root / v).resolve()
        if not candidate.is_relative_to(root):
            raise ValueError(f"path escapes workspace: {candidate}")
        if candidate.is_symlink():
            raise ValueError("refusing to write through a symlink")
        return str(candidate.relative_to(root))


# If you must have a shell tool, it is FIXED-COMMAND, never a free-form shell.
# This is OWASP's "avoid open-ended tools" as code.
_ALLOWED_COMMANDS = {"git status": ["git", "status"], "pytest": ["pytest", "-q"]}


def run_fixed_command(key: str) -> list[str]:
    if key not in _ALLOWED_COMMANDS:
        raise ValueError(f"command {key!r} is not on the allowlist")
    return _ALLOWED_COMMANDS[key]


_SQL_IDENT = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")


def build_tenant_query(table: str, tenant_id: str) -> tuple[str, list]:
    """Parameterised query with an identifier that cannot be a fragment: a table
    name cannot be parameterised, so it gets the strictest validation."""
    if not _SQL_IDENT.match(table) or table not in {"orders", "invoices"}:
        raise ValueError(f"table {table!r} is not exposed to the agent")
    return (f"SELECT * FROM {table} WHERE tenant_id = %s", [tenant_id])

Each piece maps to a named risk. path_inside_workspace is LLM10’s path traversal, and the order matters — resolve then check containment. _ALLOWED_COMMANDS is LLM03’s “avoid open-ended tools”: a shell tool is remote code execution with a friendly name. build_tenant_query is LLM10’s SQL injection, with the table name — the only non-parameterisable part — getting the strictest validation. And max_length is an LLM06 budget control.

3.4 Wiring the layers into the loop

The loop body has four obligations. Spend the step budget first, so a denied action still costs budget – otherwise an attacker gets an infinite denial loop for free. Bind every approval to action_identity(spec.name, arguments) and surface render_exact(...), the real arguments rather than the model’s description of them. And key execution on that same identity, because an approval can be double-clicked and a stream can drop: without an idempotency key, an irreversible action runs twice.

4. Numbers and trade-offs

What to do when confidence is low

Do not ask the model for a confidence score. A model’s stated probability is uncalibrated, shifts with the prompt’s wording, and is exactly the channel an attacker writes to — a payload that says “I am certain” is doing so on purpose. Low confidence must be inferred from signals the model does not author:

• risk == IRREVERSIBLE — a static property of the tool, so escalate always.
• Repeated validation failure — a deterministic counter; abort the sub-task.
• Retrieval score below threshold — a number from your own retriever; abstain or ask the user rather than generate.
• Duplicate tool-call identity — the loop detector fired; halt. A budget exhaustion terminates with a partial result, never silently.

The design rule: abstention is a first-class outcome. A system that must always answer will answer from its prior when retrieval fails — the hallucination failure mode dressed as helpfulness. Give the agent an explicit cannot_answer path, and make the harness default an unclassified action to DENY rather than ALLOW, the same fail-closed default as allowed_principals.

The honest bottom line. Every control in the first table except the decision layer is an arms race with an adversary who can read your design. The decision layer is not: a tool that cannot delete a bucket does not delete a bucket, whatever text reached the model. Spend your effort on the tool list — a static, reviewable artefact — rather than on prompt wording, which is not.

5. Pitfalls

Building the input layer first and calling it done. Symptom: a red team produces a working exploit in an afternoon and the fix is another filter. Filtering is the weakest layer and the one everyone ships first.

Gating by category, or approving a summary instead of the action. Symptom (a): reviewers approve hundreds of trivial actions and rubber-stamp the one that matters. Symptom (b): the reviewer approves “send a status email” and the executed recipient differs — invisible-character smuggling and argument substitution both produce this. Render the exact arguments and bind the approval to (tool, args).

A sandbox with egress and a mounted secret, or a broad credential behind a narrow tool. Symptom (a): exfiltration succeeds from inside the boundary and the sandbox is blamed — it is a seatbelt. Symptom (b): a read-only tool dumps the whole database, the Cursor/Supabase shape; minimise at the credential.

Writing untrusted content into memory, or into a system message. Symptom (a): an assistant behaves correctly for weeks, then acts on an instruction nobody gave it, traceable to a page summarised days earlier — memory writes need the same gate as tool calls. Symptom (b): the agent obeys text found in an issue title, because a tool result was interpolated into a system position. The most convenient API shape is the most dangerous one.

No step cap, or a non-idempotent tool behind an approval gate. Symptom (a): a runaway loop bills thousands overnight, or an attacker triggers it deliberately. Symptom (b): duplicate side effects after a double-click or a network blip — an approval is not the only retry source; transport failures are too.

6. Using it in a project

Match controls to the trifecta legs you actually have. Enumerate your tools, mark A/B/C, and let the marks drive the design — an agent with no private data and no network egress needs far less machinery than one with both. Do not start from a checklist; start from your tool list.

Reach for a sandbox when the agent executes generated code, or when a tool’s output reaches something that interprets it (a shell, a template engine, a browser). Do not use a sandbox as a substitute for least privilege: a sandbox around an over-privileged credential still has the credential.

Reach for human approval when an action is irreversible, externally visible, or value-moving. Do not gate reversible actions — you spend the reviewer’s attention before the action that matters. Reach for an input classifier when defending against opportunistic, non-adaptive abuse at scale — spam, casual jailbreaks, brand-safety violations. Do not present it as prompt-injection defence; the false confidence is worse than the missing control. Reach for a structural pattern when the task tolerates reduced generality; Plan-Then-Execute is usually the best cost/benefit.

When NOT to add guardrails at all. A read-only analytics agent with no credential beyond a public dataset and no network egress does not need an approval workflow. A prototype on synthetic data does not need PII scrubbing. The failure mode runs both ways: under-guarding ships a breach, over-guarding ships a product nobody uses.

The decision rule. For each tool, ask: if an attacker fully controlled the model’s output right now, what is the worst this tool can do? Acceptable means ship it without ceremony; otherwise fix the tool — narrow it, scope its credential, sandbox it — rather than adding a check in front of it.

7. How you would prove it worked

Guardrails are the one area where eval design is the hard part, because the obvious design measures the wrong thing. Figures below are (example).

1. Adaptive red-teaming, with the defence disclosed. The only evaluation that counts. Give the red team your full specification — prompts, classifier thresholds, allowlists — and let them optimise against it. Metric: attack success before and after each control, plus its latency and false-positive cost. Baseline: static published attack strings, which show near zero and tell you nothing. This mirrors arXiv:2510.09023.

2. Capability reduction, measured directly. Count trifecta instances in your tool list before and after. This is a static measurement — no model, no adversary — and the most reliable number in the chapter. (example) “Four tools could read private data, reach untrusted content, and send email; after splitting the mail tool and scoping the DB credential, zero could.”

3. Blast radius, then benign-task regression. For your riskiest tool, measure what a fully compromised agent can read or change — rows, dollars, hosts — before and after credential scoping; then measure completion and p95 latency with and without each control, because a control that halves attack success while halving completion is a bad trade.

8. Interview drill

“How do you stop prompt injection?” Strong answer: state first that there is no reliable detection-based fix — OWASP, NIST, and the adaptive-attack literature all agree — and that the goal is to make a successful injection non-catastrophic. Then describe capability removal: least privilege at the credential, narrow tools, complete mediation in code, a sandbox, human approval for irreversible actions, and structural patterns such as Plan-Then-Execute. Use the trifecta — private data, untrusted content, external communication, per Willison, not Anthropic — and the Rule of Two, which generalises it by adding state change, to decide which controls are mandatory. Common wrong answer: “add a prompt-injection classifier” or “tell the model to ignore injected instructions” — measured adaptive bypass above 90%.

“Why is a 95%-effective guardrail not good enough?” Strong answer: detections and exposures compound the other way. If each of $N$ independent exposures is caught with probability $d$, the probability at least one gets through is $1-d^N$. At $d=0.95$ and $N=20$ documents, that is 64%. The exponent, not the classifier, is the problem — so the fix is fewer exposures and less impact. Common wrong answer: “95% means one in twenty gets through, which is acceptable.” Per document, not per task.

“Which actions need human approval, and where does a sandbox belong?” Strong answer: approval by consequence — gate what is irreversible, externally visible, or value-moving, and auto-approve the rest, because approvals cost $g\cdot n_a\cdot t_r$ and reviewer attention is finite. Bind each approval to the exact (tool, args) identity and render the real arguments. A sandbox complements this by removing the [B] and [C] legs; the distinction that matters is the boundary type (containers share the host kernel, gVisor mediates syscalls, Firecracker gives hardware virtualisation), and a sandbox with network egress and a mounted secret is a seatbelt. Common wrong answers: “gate all destructive actions” with no criterion, and “we run the code in Docker”, which answers kernel isolation and says nothing about egress, mounts, or credentials.

“Which OWASP entries matter most for agents?” Strong answer: LLM01 Prompt Injection for the input boundary and LLM03 Excessive Agency for the consequence — OWASP frames them as the input-side compromise and the agency-side amplification. Note the 2026 renumbering: Excessive Agency moved from LLM06 to LLM03, and System Prompt Leakage became LLM08 Hidden Context Exposure. Common wrong answer: quoting the 2025 numbering as current.

Next: The RAG Pipeline — retrieved chunks are the most common indirect-injection delivery surface, which is why the analysis above is a prerequisite for reasoning about a RAG system’s threat model.