Intent Recognition and Routing

TL;DR — A router picks the pipeline before any work is done: a rule match, then a small classifier, then an LLM call — plus the option people forget, ask the user. The escalation threshold is not a magic number but the solution of a cost inequality in which the cost of a wrong route sits in the denominator, so it is only meaningful once confidence is calibrated. The metric that matters is per-class precision: a wrong route is unrecoverable, so the router multiplies every handler’s accuracy instead of adding to it.

1. Why this exists

You ship a support assistant. Version one sends every message to a frontier model with the full tool list attached. It works, and then two things happen. The bill arrives with the traffic, and someone measures p95 latency and finds that 60% of the traffic is “where is my order”, “what plan am I on”, “reset my password” — three deterministic lookups a regex and a database read answer in 20 ms.

So you add a router. A small model looks at the message, picks one of eleven intents, and the pipeline branches. Now the failure mode is worse than the cost problem it fixed:

A customer writes “I was charged twice for the same month”. The router classifies it as cancel_subscription with probability 0.71 — above your threshold of 0.7 — and the cancel handler dutifully cancels the account. The user never asked for that. There is no error in any log; every stage reported success. Un-cancelling is a support ticket, a refund, and an apology.

That example is the whole chapter. Two properties of routing follow from it:

A wrong route is not a wrong answer, it is a wrong world. An answer can be corrected next turn; a route commits a downstream process whose own quality is then irrelevant, because it was correct about the wrong question. Routing is therefore a classification problem with an asymmetric decision problem on top, and accuracy is the wrong headline metric.

The router is a multiplier, not an addend. At 95% router accuracy and four 95% handlers, the end-to-end success rate is not 95% and not 90%: under independence it is $0.9 5^{5} \approx 0.77$ . The only structural escape is to let the router abstain, because an abstention is not multiplied by anything.

Routing is also where the decision ladder is most concrete. Rung 1 (rules) handles deterministic commands; rung 4 (fine-tuning) buys a 20 ms classifier; rung 2 (a prompt) is the fallback. Most teams skip rung 1 because it feels unglamorous, then pay an LLM call to match a five-character reset command.

2. How it works

2.1 Four tiers, cheapest first

flowchart TD
    M["message"] --> R{"rule matches?"}
    R -->|yes| H0["deterministic handler<br/>(regex, ID lookup, command)"]
    R -->|no| C["classifier<br/>label + calibrated confidence p"]
    C --> B{"|p1 - p2| < margin?"}
    B -->|yes| Q["clarify:<br/>ask the user"]
    B -->|no| T{"p >= tau?"}
    T -->|yes| H1["intent handler<br/>(small model / template)"]
    T -->|no| L["LLM router / general handler"]
    L --> H2["intent handler or free-form answer"]
    H0 --> O["response"]
    H1 --> O
    Q --> O
    H2 --> O

The tiers are ordered by a single quantity: the cost of being wrong, not the cost of running them. A rule matching a bare ticket <digits> message cannot be fuzzily wrong — either the message is a ticket id or it is not — so it goes first even though it is also the cheapest. A classifier is probabilistically wrong, so it is gated by a threshold. The LLM is expensive but broad, so it absorbs the long tail.

The clarify branch is a real action, not a failure. Asking “did you mean cancel your subscription, or get a refund for a duplicate charge?” costs the user five seconds; routing it wrong costs a cancellation. Put it in the topology from the start, because forcing every message into one of eleven labels is what makes the “charged twice” example inevitable.

2.2 The threshold is the solution of a cost inequality

Let a message arrive and let the classifier output a label $\overset{y}{^}$ with confidence $\overset{p}{^} \in [0, 1]$ . Assume for now that $\overset{p}{^}$ really is the probability that $\overset{y}{^}$ is correct; §2.3 is about making that assumption true. Define three costs, all measured in the same unit — pick one and stay in it, because the whole point is that a millisecond of latency has to be comparable to a support ticket:

$C_{s}$ — the cheap path: compute plus the latency you charge yourself for.
$C_{l}$ — the escalation path, which dominates in a real system: LLM tokens, prefill latency, and any tool round-trips. By construction $C_{l} > C_{s}$ .
$C_{e}$ — a wrong route: downstream damage plus the user’s recovery effort. Everyone omits this term, and omitting it is what produces thresholds picked by vibes.

Dispatching on the cheap path is correct with probability $\overset{p}{^}$ and wrong otherwise, so its expected cost is

$E [cheap] = C_{s} + (1 - \overset{p}{^}) C_{e}$

Escalating is assumed correct — and this assumption is a debt, because if the LLM router is itself 90% accurate you have merely moved the error one hop. Its expected cost is

$E [escalate] = C_{l} + (1 - a_{l}) C_{e}$

with $a_{l}$ the LLM router’s accuracy. In the common case where $a_{l}$ is close to 1 the second term is dropped, and the two lines give the decision rule:

$dispatch cheap ⟺ \overset{p}{^} > τ, τ = 1 - \frac{C _{l} - C _{s}}{C _{e}}$

Read the threshold in three pieces.

The numerator $C_{l} - C_{s}$ is the premium you pay for the better path. It is what you save by being right cheaply. If escalation were free ( $C_{l} = C_{s}$ ), the premium is zero, $τ \to 1$ , and you should escalate everything — which is the honest description of a system with no router.
$C_{e}$ in the denominator is what a mistake costs, and it moves $τ$ in the direction you would not guess from the formula. A large $C_{e}$ shrinks the fraction toward zero and pushes $τ$ toward 1, so you escalate more. A cheap, reversible error — a wrong FAQ article the user ignores — has a small $C_{e}$ , the fraction grows, $τ$ falls, and the router serves more traffic itself.
The fraction is dimensionless, so both sides must be in the same unit. If you measure the premium in dollars and the error in “user annoyance”, the number you derive is meaningless. Pick the unit — I use cents of total cost of ownership — and convert latency and human minutes into it explicitly.

Two consequences are counter-intuitive and worth memorising:

Situation	$τ$	Behaviour
$C_{e} ≫ C_{l}$ (a mistake is catastrophic)	close to 1	escalate almost everything; the router earns nothing
$C_{e} ≪ C_{l} - C_{s}$ (error cheaper than the premium)	$\leq 0$	always take the cheap path; the classifier is dead weight
$C_{e} \approx C_{l} - C_{s}$	around 0.5	the regime where a router actually pays

The middle row is the interview answer: if a wrong route costs less than the escalation premium, a router makes your life harder, and the effort belongs in the fallback message instead.

2.3 Calibration: a softmax is not a probability

$τ$ is a probability threshold, so $\overset{p}{^}$ has to be a probability. The softmax output of a modern neural network overestimates its own accuracy — a phenomenon documented in Guo et al., On Calibration of Modern Neural Networks (ICML 2017), which found that deeper, wider, and batch-normalised networks are worse calibrated than the shallower networks of a decade earlier. A classifier that reports 0.9 while being right 70% of the time makes $τ = 0.9$ behave like $τ = 0.7$ : it serves messages it should have escalated, and nothing in the logs looks abnormal.

The standard repair is temperature scaling, a one-parameter change to how the probabilities are formed. With logits $z_{k}$ for class $k$ :

$\overset{p}{^} = k max \frac{exp ( z _{k} / T )}{\sum _{j} exp ( z _{j} / T )}$

$z_{k}$ — the raw score for class $k$ . Its scale is arbitrary; only its ordering is trained.
$T$ — the temperature. $T = 1$ leaves the softmax as trained; $T > 1$ flattens it, which is what an overconfident network needs. It is the one parameter fitted after training, on held-out data, by minimising negative log-likelihood. It cannot change which label wins, so it changes only the threshold behaviour — exactly what routing depends on, and the reason $τ$ and $T$ must be fitted together rather than one after the other (§3 shows a fitted $T$ silently disabling both gates).
The $max_{k}$ exists because the router needs a scalar for a multi-class decision. It is lossy: the full distribution holds the margin information used by the clarify branch.

Whether the fitted $\overset{p}{^}$ is any good is measured by expected calibration error: bin by confidence, compare each bin’s mean confidence with its accuracy.

$ECE = b = 1 \sum B \frac{∣ S _{b} ∣}{N} acc (S_{b}) - conf (S_{b})$

$B$ — the number of confidence bins (10 equal-width bins is the convention).
$S_{b}$ — the predictions whose confidence falls in bin $b$ . $∣ S_{b} ∣/ N$ is the weight; removing it lets a bin with four examples move the number as much as one with four thousand.
$acc (S_{b})$ and $conf (S_{b})$ — that bin’s empirical accuracy and its mean reported confidence. The absolute value discards the sign, and always-overconfident and always-underconfident need opposite fixes.

ECE is a footgun of its own: it depends on $B$ and is not a proper scoring rule, so two models can share an ECE with different usefulness. Report it beside the coverage numbers, never alone.

2.4 Precision is the safety metric, recall is the coverage metric

For a single route $c$ , over the traffic the router dispatched to $c$ :

$prec_{c} = \frac{T P _{c}}{T P _{c} + F P _{c}}, rec_{c} = \frac{T P _{c}}{T P _{c} + F N _{c}}$

$T P_{c}$ — messages that truly belong to $c$ and were dispatched to $c$ . A message that belongs to $c$ but was escalated is not a true positive for the cheap route; it is a missed saving.
$F P_{c}$ — dispatched to $c$ , belongs elsewhere. Each is a committed wrong action: the cancellation.
$F N_{c}$ — belongs to $c$ , went elsewhere. If that elsewhere was the escalate branch the message is not lost, just more expensive.

That asymmetry is why the two are not interchangeable. Precision bounds the damage; recall bounds the cost. A system optimising accuracy alone trades them against each other invisibly, usually by over-firing the majority class: on a router whose traffic is 80% order_status, aggregate accuracy is a metric about order_status. Report macro averages (each intent weighted equally) on the dashboard and keep the micro number for capacity planning — the class that fires once a day is the one whose FP nobody noticed until a customer complained.

2.5 Abstention does not compound

If a pipeline has $m$ stages and stage $i$ is correct with probability $a_{i}$ , end-to-end success is $\prod_{i = 1}^{m} a_{i}$ under the (optimistic) assumption of independent failure. At $a = 0.95$ and $m = 5$ that is $0.774$ .

Add one edge — the router escalates with probability 0.05 to an LLM that is 97% correct — and the arithmetic becomes $0.95 \times 0.95 + 0.05 \times 0.97 = 0.951$ , not $0.90$ . Confidence became a gate instead of a multiplier, which is the strongest argument for spending engineering time on the abstain path rather than the classifier’s last two points of accuracy.

3. Implementation

The listing is the routing policy plus its evaluation harness: runnable on Python 3.11+ with the standard library only. The classifier is a lexical scorer standing in for the real one; everything around it (rule tier, calibration hook, threshold, clarify band, cost model, per-class accounting) is the part to copy.

# router.py -- Python 3.11+, standard library only.
import math
import re
from dataclasses import dataclass

# --- tier 0: rules. A rule must be exact, or it becomes the bug.
RULES = [
    (re.compile(r"\s*ticket\s*#?\s*\d{3,}\s*", re.I), "ticket_status"),
    (re.compile(r"\s*/(reset|help)\s*", re.I), "smalltalk"),
]

# --- tier 1: a stand-in for the encoder classifier. A real one returns logits
# from a fine-tuned encoder; this one sums keyword weights. Same interface.
LEXICON = {
    "order_status":   {"order": 1.0, "package": 1.0, "shipped": 0.9, "track": 1.0, "arrived": 0.8},
    "cancel":         {"cancel": 1.0, "stop": 0.6, "unsubscribe": 1.0, "account": 0.3},
    "refund":         {"refund": 1.0, "money": 0.8, "charged": 0.7, "billing": 0.4},
    "plan_lookup":    {"plan": 1.0, "which": 0.3, "tier": 0.6, "card": 0.3},
    "profile_update": {"address": 1.0, "email": 1.0, "update": 0.8, "change": 0.8},
    "__oos__":        {"weather": 1.0, "flight": 0.9, "reminder": 1.0, "resume": 1.0},
}


class LexicalClassifier:
    def __init__(self, temperature=1.0):
        self.temperature = temperature

    def predict(self, text):
        toks = re.findall(r"[a-z0-9]+", text.lower())
        logits = {lab: sum(w.get(t, 0.0) for t in toks) for lab, w in LEXICON.items()}
        m = max(logits.values())
        exps = {k: math.exp((v - m) / self.temperature) for k, v in logits.items()}
        total = sum(exps.values())
        probs = {k: v / total for k, v in exps.items()}
        label = max(probs, key=lambda k: probs[k])
        return label, probs[label], probs

    def fit_temperature(self, held_out):
        """Fit T by minimising NLL. See the pitfall in section 5: with a small
        held-out set and few errors this runs to the edge of the grid."""
        best_t, best_nll = 1.0, float("inf")
        for t in (0.1, 0.2, 0.3, 0.5, 0.75, 1.0, 1.5, 2.0, 3.0, 5.0):
            self.temperature = t
            nll = 0.0
            for text, gold in held_out:
                _, _, probs = self.predict(text)
                nll -= math.log(max(probs.get(gold, 0.0), 1e-12))
            if nll < best_nll:
                best_t, best_nll = t, nll
        self.temperature = best_t
        return best_t


@dataclass(frozen=True)
class Decision:
    route: str      # an intent label, or "__llm__", or "__clarify__"
    tier: str       # "rule" | "classifier" | "llm" | "clarify"
    score: float    # calibrated P(route is correct); 1.0 for a rule


class Router:
    def __init__(self, classifier, ask_llm, tau, margin=0.0):
        self.clf, self.ask_llm, self.tau, self.margin = classifier, ask_llm, tau, margin

    def route(self, text):
        for pattern, label in RULES:                       # tier 0, always first
            if pattern.fullmatch(text):
                return Decision(label, "rule", 1.0)
        label, conf, probs = self.clf.predict(text)         # tier 1
        ranked = sorted(probs.values(), reverse=True)
        if self.margin and len(ranked) > 1 and ranked[0] - ranked[1] < self.margin:
            return Decision("__clarify__", "clarify", conf)
        if conf < self.tau:                                # tier 2
            return Decision(self.ask_llm(text), "llm", conf)
        return Decision(label, "classifier", conf)


def evaluate(router, data, c_small, c_large, c_error, c_clarify, llm_acc):
    """Coverage, escalation rate, per-class P/R and cost, on a labelled set."""
    per_class, cost = {}, 0.0
    escalated = clarified = cheap = 0
    for text, gold in data:
        d = router.route(text)
        if d.tier == "llm":
            escalated += 1
            cost += c_large + (1.0 - llm_acc) * c_error
            continue
        if d.tier == "clarify":
            clarified += 1
            cost += c_clarify
            continue
        cheap += 1
        correct = d.route == gold
        cost += c_small + (0.0 if correct else c_error)     # the C_e term
        for lab in {gold, d.route}:                          # keep every key present
            per_class.setdefault(lab, [0, 0, 0])
        tp, fp, fn = per_class[d.route]
        per_class[d.route] = [tp + (1 if correct else 0), fp + (0 if correct else 1), fn]
        if not correct:
            tp, fp, fn = per_class[gold]
            per_class[gold] = [tp, fp, fn + 1]
    n = len(data)
    return {
        "coverage": cheap / n,
        "escalation_rate": escalated / n,
        "clarify_rate": clarified / n,
        "cost_per_1k": 1000.0 * cost / n,
        "per_class": {
            k: {"precision": (v[0] / (v[0] + v[1])) if v[0] + v[1] else 0.0,
                "recall": (v[0] / (v[0] + v[2])) if v[0] + v[2] else 0.0}
            for k, v in per_class.items()
        },
    }

Driving it with two labelled lists — 14 messages to fit the temperature, 12 to evaluate — and a stub ask_llm that answers __oos__:

HELD_OUT = [("where is my package", "order_status"), ("cancel my plan", "cancel"),
            ("refund please", "refund"), ("what plan do i have", "plan_lookup"),
            ("change my address", "profile_update"), ("money back please", "refund"),
            ("cancel my subscription now", "cancel"), ("how do i update my address", "profile_update"),
            ("can you write my resume", "__oos__"), ("what time is it in berlin", "__oos__"),
            ("set a reminder for 6pm", "__oos__"), ("please stop my plan", "cancel"),
            ("my email is wrong", "profile_update"), ("do i have a card on file", "plan_lookup")]
EVAL = [("where is my order", "order_status"), ("has my package shipped", "order_status"),
        ("i want a refund", "refund"), ("money back please", "refund"),
        ("cancel my subscription now", "cancel"), ("what plan am i on", "plan_lookup"),
        ("how do i update my address", "profile_update"), ("ticket 48213", "ticket_status"),
        ("what is the weather tomorrow", "__oos__"), ("book me a flight to tokyo", "__oos__"),
        ("can you write my resume", "__oos__"), ("please stop my plan", "cancel")]

clf = LexicalClassifier()
print("fitted temperature T =", clf.fit_temperature(HELD_OUT))    # prints 0.2
clf.temperature = 1.0                       # start from the raw confidences
for tau in (0.30, 0.40, 0.50, 0.60):
    m = evaluate(Router(clf, lambda t: "__oos__", tau=tau, margin=0.20), EVAL,
                 c_small=0.02, c_large=1.00, c_error=8.00, c_clarify=0.35, llm_acc=0.97)
    print(f"tau={tau:.2f} coverage={m['coverage']:.2f} "
          f"escalation={m['escalation_rate']:.2f} cost/1k={m['cost_per_1k']:.0f}")

Sweeping $τ$ on the 12 evaluation messages (example — this is the printed output of the listing above, on a synthetic set, not production data):

$τ$	coverage	escalation	clarify	cost / 1k msgs	worst routed precision
0.30	0.75	0.00	0.25	102	1.00
0.40	0.25	0.50	0.25	712	1.00
0.50	0.25	0.50	0.25	712	1.00
0.60	0.08	0.67	0.25	916	1.00

Precision is 1.00 in every row, because the one message the classifier gets wrong ("please stop my plan" -> plan_lookup) has a 0.105 probability margin and the clarify band is 0.20 wide. So precision on the routed set cannot choose the threshold — it is a constant here. Only the cost column can, and it moves by roughly 9x across a range that looks like a small tuning tweak.

Now change one thing: use the temperature that was fitted, $T = 0.2$ . The confidences sharpen past every threshold and past the clarify margin:

Configuration	coverage	escalation	clarify	cost / 1k	worst routed precision
$T = 1.0$ , $τ = 0.30$ , margin 0.20	0.75	0.00	0.25	102	1.00
$T = 0.2$ (fitted), $τ \in {0.30, 0.40, 0.60}$	1.00	0.00	0.00	687	0.50

Same router, same threshold, same data. The temperature fit did what it was asked — it maximised likelihood on held-out data — and silently disabled both safety gates, because a threshold expressed in probability units has to be re-derived after the probabilities move. That is the operational form of the warning in §2.3, and it is why the escalation rate belongs on the dashboard: it is the only number in this table that would have told you.

The version you should actually ship

The classifier is the only part not to hand-roll. A prompt-free few-shot approach that runs in milliseconds on CPU is SetFit (Tunstall et al., Efficient Few-Shot Learning Without Prompts), fine-tuning a sentence encoder contrastively and then fitting a linear head:

# pip install setfit==1.1.0  (as of 2026-10; check the release you install)
from setfit import SetFitModel, Trainer, TrainingArguments
from datasets import Dataset

ds = Dataset.from_dict({"text": texts, "label": [LABELS.index(y) for y in labels]})
model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")
Trainer(model=model, train_dataset=ds,
        args=TrainingArguments(batch_size=16, num_epochs=1)).train()
model.save_pretrained("router-v3")

Two details matter more than the library choice. The training set needs an explicit out-of-scope class — CLINC150 has one precisely because Larson et al. found that classifiers assuming every query is in scope score well on in-scope traffic and fail on everything else. And the confidence you threshold must come from the fitted head, not a cosine similarity.

If the classifier is an LLM rather than an encoder, constrain the decode so the label is structurally guaranteed: vLLM’s guided_choice does exactly this (extra_body={"guided_choice": [...]}, vLLM v0.8.3 docs), and guided_json does the same against a schema. Note the measured cost of over-constraining: Tam et al. found reasoning benchmarks degrade under strict output formats, so constrain the label field and leave the reasoning free-form rather than forcing the whole response through JSON.

4. Numbers and trade-offs

The datasets you would sanity-check on

Dataset	Size	Classes	Notes
CLINC150 (arXiv:1909.02027)	22,500 in-scope + 1,200 out-of-scope queries	150 intents, 10 domains	the standard OOS benchmark; 100 train / 20 val / 30 test per intent, and 1,000 OOS test queries
Banking77 (arXiv:2003.04807)	13,083 utterances	77 intents, single domain	all in-scope; near-duplicate utterances make a random split look better than it is
MASSIVE (arXiv:2204.08582)	1M utterances	60 intents, 55 slots, 18 domains, 51 languages	parallel across languages; the one to use if you route non-English traffic

Two facts shape designs: CLINC150 is single-intent, so it says nothing about the multi-intent messages real users send; and Banking77’s 77 intents are one domain, so it measures fine-grained discrimination rather than breadth.

What a small model buys you

SetFit’s reported results: with 8 labelled examples per class on Customer Reviews it is competitive with full-set fine-tuning, despite the fine-tuned model being three times larger; training is an order of magnitude faster than PEFT/PET, and its MiniLM variant two orders of magnitude faster than T-Few for an average 3.1 accuracy-point reduction. Storage is 70 MB (MiniLM) or 420 MB (MPNet) against 11.4 GB for T-Few’s T0-3B. The trade-off named is accuracy per gigabyte of serving footprint, and for a router seeing 10k messages a day it is the right one.

If the “cheap path” is itself an LLM call, see RouteLLM: preference-trained routers cut cost over 2x in some settings without compromising quality, and transfer to a new strong/weak pair at test time — so you do not retrain when the vendor ships.

The latency budget

(example) The latency shapes, in the same unit as §2.2: a rule under 1 ms, a batched CPU encoder classifier 5-20 ms, a cross-encoder rerank of the top 3 intents +20-60 ms, an LLM router 300-1500 ms to first token, a clarify question one user turn. Note the classifier is never the latency problem: insert one but still call the LLM whenever it is unsure, and you have added 20 ms to every message and saved nothing.

5. Pitfalls

Thresholding an uncalibrated softmax. Symptom: the escalation rate drifts away from what you designed, and it drifts with the input distribution rather than staying fixed. A confidence of 0.8 means “0.8 in the training distribution”; a demo with different phrasing shifts the whole distribution. Fix: fit $T$ on held-out data, and monitor the escalation rate as a metric in its own right — it is the cheapest alarm you will ever install.

No out-of-scope class at training time. Symptom: OOS messages are routed with high confidence to whichever intent has the highest lexical overlap — “book me a flight” becomes order_status. CLINC150’s contribution is that this is the default behaviour of classifiers trained without OOS examples. Fix: train the OOS class on the domain’s real long tail, and keep the LLM fallback for the part you cannot enumerate.

Aggregate accuracy hiding a dead route. Symptom: overall accuracy 94%, but one intent has precision 0.4 and nobody noticed because it is 2% of traffic. This is the failure the router exists to prevent, committed by its own dashboard. Fix: macro-averaged per-class precision first on the dashboard.

Error compounding through the route. Symptom: every component’s metrics look fine and end-to-end task completion is far below their product. The gap is messages routed confidently and wrongly, which never reach the retry or clarify path. Fix: put route, confidence and downstream outcome on one trace, and make task success the headline number rather than routing accuracy.

Rules that are not exact. Symptom: a rule fires on 30% of traffic and the intent it maps to becomes useless. A rule has no confidence to gate on, so a fuzzy rule is more dangerous than the classifier it protects. Fix: a precision test on a labelled slice in CI for every rule.

Building the router before the taxonomy is stable. Symptom: retraining every sprint and a growing pile of relabelled data as intents split and merge. A classifier encodes a taxonomy, and a moving taxonomy is not ready to be encoded. Fix: ship rules plus the LLM fallback, log, and derive the taxonomy from the logs rather than a whiteboard.

6. Using it in a project

Reach for a router when you have high-frequency, distinguishable intents, an asymmetric cost of misrouting, and enough labelled examples to measure per-class precision. The paying traffic shape is a heavy head and a long tail: the head is served at 20 ms and dominates the cost curve, the tail at LLM latency.

Do not reach for a router when the mapping is deterministic. A command grammar (/reset, /export --format csv), a ticket-number regex, or an intent field the client already sends are all rung 1 — and an LLM router at 97% on them is worse than a parser at 100%, because it attaches a wrong-route cost. The test: can you write the specification as a decision table? If yes, write the decision table.

Three more cases where the router is the wrong layer:

The “intent” is really retrieval. If every branch ends in the same RAG call with a different filter, you have a metadata-filtering problem, not an intent problem. Let the retriever’s score do the work.
The downstream call is already cheap. A 100-token handler makes the premium $C_{l} - C_{s}$ small next to any real $C_{e}$ , so $τ$ is high and you escalate almost everything. You built a classifier that never fires.
The taxonomy changes weekly. Ship the LLM fallback and log instead; a router trained on a moving taxonomy generates confident wrong routes.

7. How you would prove it worked

Three metrics, one baseline, one ablation. The metrics are not interchangeable — pick the one that matches the claim you are making.

Safety: macro-averaged per-class precision of the routing decision, for the classes the classifier serves. It bounds the cancellation incidents.
Economics: coverage at the deployed $τ$ and the resulting cost per 1,000 messages, computed with $C_{s}, C_{l}, C_{e}, C_{c l a r i f y}$ written down before the experiment.
Outcome: end-to-end task completion, with the escalation rate as the leading indicator.

Baseline: the always-LLM configuration that shipped before the router, on the same traffic sample. Without it, “the router is accurate” refers to nothing.

Ablation: remove one tier at a time. Removing the rules tier says how much of the head actually needed a model; removing the clarify band says whether the ambiguous messages were misrouted or correctly guessed; removing the OOS class prices your optimism.

Reporting template (example — replace every number). “Against the always-LLM baseline, cost per 1,000 messages fell from $14.20 to $3.10 (example) and p95 latency from 2.4 s to 380 ms (example), at equal end-to-end task completion (78% vs 77%, example). Macro precision on routed classes is 0.96 and coverage is 0.61 at $τ = 0.55$ . It still misroutes multi-intent refund-and-cancel messages, which the clarify band was meant to catch; the fix is to detect conjunctive phrasing before classification rather than after.”

8. Interview drill

“How do you choose the confidence threshold?” Strong answer: derive it. Dispatch cheap iff $\overset{p}{^} > 1 - (C_{l} - C_{s}) / C_{e}$ , the premium for escalating over the cost of a wrong route. Then the two consequences: if a mistake is cheap relative to the premium, always take the cheap path; if it is catastrophic, the router never fires and should not exist. Common wrong answer: “we looked at the precision-recall curve and picked 0.9.” That answers a question about the classifier, not the decision, and contains no cost — so it cannot tell a wrong FAQ answer from an unauthorised cancellation.

“Your classifier says 0.92. What does that mean?” Strong answer: nothing until it is calibrated. Modern nets are overconfident, so 0.92 may mean 0.75 empirical accuracy; fit temperature scaling on held-out data and verify with expected calibration error, because the threshold is expressed in probability units. Trap: “it means 92% likely to be right” — that is the definition of calibration, not a property of softmax outputs.

“When would you not use an LLM router?” Strong answer: when the task is deterministic — a command grammar, an ID lookup, a client-supplied intent field. 97% on a decision table is worse than a parser at 100%, because it adds a wrong-route cost. Also when the downstream call is already cheap: the escalation premium is small and $τ$ collapses. Common wrong answer: “when latency matters.” That is true of every router; the distinguishing question is whether the mapping is specifiable.

“The router is 95% accurate and the handlers are 95% accurate. Is the system 95% accurate?” Strong answer: no. If a wrong route is unrecoverable the router is a multiplier: roughly $0.9 5^{5} \approx 0.77$ across five stages. The escape is abstention, which takes the message out of the product into a recoverable path.

Next: NL2SQL (Text-to-SQL) — the same shape of decision, one layer down: choosing a schema element is a commitment whose errors show up as confident, valid, wrong SQL.

Keyboard shortcuts

Henry's Notebook