Model Router

Read the transcript

1. Not the gateway’s kind of routing

Host: So today we’re talking about model routing, but I want to be really precise about what we mean, because the word gets used for two totally different things. There’s the async-ai-gateway kind of routing, which is about failover — you’ve got replicas of the same provider, and you route to whichever one is actually healthy right now. That’s a reliability problem.

Guest: Right, and that’s not what we’re doing here. Model Router isn’t asking ‘which replica is up’ — it’s asking ‘does this task even need the bigger model, or would a cheaper, faster model handle it just fine.’ It’s matching task difficulty to model capability and price, not swapping between equivalent things. Same word, routing, almost no shared logic — which is exactly why we keep these labs apart, so each one stays about one clean problem.

2. The finding: 19% more correct, 11× worse per correct

Host: Okay so let’s get to the number that started this whole lab. You told me the first version of your test suite actually asserted the conventional wisdom — escalation saves money, keeps quality — and it failed. What did the failure look like?

Guest: Two hundred tasks, and we gave the policy the best possible confidence signal — low exactly when the answer is wrong, which no real system gets. Cheap-only gets 166 correct for 0.04 in total cost. Escalating gets 197 correct, but total cost jumps to 0.55. Do the division and cost per correct answer goes from 0.00024 to 0.00279 — accuracy up 19%, cost per correct answer eleven times worse.

Host: Eleven times worse feels like it must be about that huge price gap between the tiers though — isn’t that just because the expensive model is 75 times pricier?

Guest: That’s the natural assumption, but no — the relationship holds at 10x, 5x, even 2x. It’s arithmetic, not pricing. The cheap model is already right 83% of the time, so it’s throwing off a huge pile of nearly-free correct answers; escalation only adds a smaller batch of expensive correct ones on top, and averaging those together can only push the cost-per-correct number up. That’s why cost per correct answer is a trap here — it’s the metric everyone instinctively reaches for, and it’s built to hide exactly this.

3. The number that actually decides, and everything else the tests won’t let slide

Host: So if cost per correct is a trap, what’s the number you’d actually put in front of someone deciding whether to escalate?

Guest: Marginal cost per rescued answer — escalated cost minus baseline cost, divided by escalated correct minus baseline correct. That isolates what the extra correctness actually cost you: about 0.00044 at a 2x price ratio, about 0.016 at 75x. Whether that’s worth paying isn’t a metrics question anymore, it’s a product question — what does a wrong answer cost you — and the router is right to refuse to answer it for you.

Host: That’s a nice hinge point. Before we close out, walk me through the rest of what’s pinned down — the smaller behaviors that don’t get the spotlight but still matter.

Guest: Capability gets filtered before cost, so the cheapest model in the fleet never gets picked for work it literally can’t do just because it’s cheap. If nothing clears the quality floor the router still answers, but it says so in the reason string instead of failing silently, and a budget cap raises the tier rather than quietly downgrading it without an alert. Ties break on name so runs are reproducible, and a bad confidence signal produces code that looks completely fine — same escalation, same double billing, far less benefit — which is exactly why it has to be measured rather than trusted. That leaves the real open questions for whoever owns this in production: what a wrong answer is worth, who keeps that quality number honest as it drifts, whether a model’s capability claims can be trusted, and that escalation spends latency too, not just money, so a task that takes two round trips can miss a deadline the single expensive call would have met.

Not covered

The planner wanted these and found nothing in the source to support them:

A concrete dollar figure for what a wrong answer costs in a real product — the excerpts raise this as the deciding question but deliberately leave it unanswered
A worked example of the marginal-cost formula changing a real team’s escalation decision end-to-end

Generated from this page by Claude Sonnet 5 on 2026-08-23, spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

Module 4 argues that model choice is a routing decision and that the cost-latency-quality triangle should be measured rather than assumed. This lab does the measuring, and the answer was not what the lab was originally written to show.

Source: labs/model-router

Not the same routing as the gateway

async-ai-gateway routes between replicas of equivalent providers by observed success rate — a health problem. This lab routes between models of different capability and price by what the task needs. Both are called routing; they share almost no logic, and the labs are kept apart so each stays about one thing.

The path a task takes

flowchart TD
    Task["Task<br/>task_class · estimated_tokens"] --> Cap{"Which models declare<br/>support for this task class?"}
    Cap -->|None| Err["NoCapableModel<br/>raised, not guessed"]
    Cap -->|"Capable set, cheapest first"| Pol{"Policy"}

    Pol -->|CheapestCapable| P1["First in the capable set"]
    Pol -->|QualityFloor| P2["Cheapest clearing the floor<br/>or best available, reason recorded"]
    Pol -->|BudgetCapped| P3{"Above the ceiling?"}
    P3 -->|Yes| Budget["BudgetExceeded<br/>raised, never a silent downgrade"]
    P3 -->|No| P1

    P1 --> Run["Run on the chosen model"]
    P2 --> Run

    Run --> Conf{"Confidence below threshold?"}
    Conf -->|No| Done["Answer · one call billed"]
    Conf -->|"Yes, and a better model exists"| Esc["Run best model too<br/>BOTH calls billed"]
    Conf -->|"Yes, already on the best"| Done
    Esc --> Marginal["Answer · two calls billed<br/>the rescued answer has a price"]

The finding

Escalation — try the cheap model, fall back to the expensive one when it looks unsure — is standard advice, and the intuition is that it saves money while holding quality. The first version of this lab’s test suite asserted exactly that, and it failed.

Over a fixed 200-task workload, with a perfect confidence signal — low exactly when the answer is wrong, the best case any escalation policy could hope for:

	Cheap only	Escalating
Correct	166 / 200	197 / 200
Total cost	0.040	0.550
Cost per correct answer	0.00024	0.00279

Accuracy up 19%. Cost per correct answer 11× worse. And it is not an artefact of the 75× price gap between those tiers — the relationship holds at 10×, 5×, and 2×.

The cause is arithmetic rather than modelling. A cheap model already right 83% of the time contributes a large number of cheap correct answers; escalation adds a smaller number of expensive ones, so the average can only move upward. Cost per correct answer is a trap metric here, and it is the one everybody reaches for.

The number that actually decides

What escalation buys is marginal correct answers, so the honest question is what each rescued answer cost:

marginal cost per rescued answer
  = (escalated_cost − baseline_cost) / (escalated_correct − baseline_correct)

About 0.00044 per rescued answer at a 2× price ratio; about 0.016 at 75×. Whether either is worth paying depends on what a wrong answer costs you — a product question, and one the router should not pretend to answer. test_marginal_cost_per_rescued_answer_scales_with_the_cost_ratio pins the relationship so the conclusion cannot quietly invert.

What else the tests pin

Capability filtering happens before cost. small is the cheapest model in the fleet and cannot write code. A router sorting by price first picks it and is cheap, fast, and wrong.

A quality floor nothing clears falls back visibly. The router still returns a decision — you cannot serve nothing — but records in the reason string that the floor was not met, so a bar nobody is hitting is auditable rather than invisible.

A budget cap raises instead of downgrading. Silently choosing a worse model to stay under budget is a quality regression with no alert attached.

Ties break on name, not registration order. A router whose choice depends on dict iteration produces cost reports nobody can reconcile between runs.

The confidence signal’s usefulness is invisible in the API. Swapping an informative signal for a meaningless one changes nothing a type checker or a code review would catch — escalation still fires, still bills twice, and returns far less. That is why it has to be measured.

Run it

cd labs/model-router
uv venv .venv && uv pip install --python .venv/bin/python -e '.[dev]'
./.venv/bin/python -m ruff check . && ./.venv/bin/python -m mypy src && ./.venv/bin/python -m pytest -q

16 tests, ruff clean, mypy --strict clean.

Principal-level discussion points

What is a wrong answer worth? Until that number exists, no escalation policy can be justified or rejected — and most teams adopt one without ever asking.
Who owns ModelSpec.quality? It is a static claim here. In production it decays, and a router optimising against stale quality numbers degrades silently.
Capability declarations are trust. A model that claims it can reason and cannot will be routed work it fails at, cheaply and confidently.
The triangle’s third corner. Escalation spends latency as well as money. A task that escalates takes two round trips and may miss a deadline the single expensive call would have met.

Architecture: Task-Based Model Routing — the design review, where the escalation finding above becomes a cost argument.
Module 4: AI Infrastructure — the cost-latency-quality triangle, and why model choice is a routing decision.
Async AI Gateway — health-aware routing between equivalent providers, which is a different problem.