Architecture: Calibrated Decision Tier

Read the transcript

1. The bounded question nobody should pay generation prices for

Host: So here’s the thing that bugs me about most AI spend: a huge chunk of what gets sent to a model isn’t really generation at all. It’s a bounded question — which queue does this go to, is this request in scope, is this safe to act on — and you’re paying generation’s latency and price for what is basically a classification call, then parsing text back into an answer anyway.

Guest: Right, and a decision model skips that whole charade — it answers the bounded question directly from a set you define, and gives you a probability over those answers instead of prose to parse. That probability is the entire design: act on the confident ones, escalate the rest, and the expensive model only ever sees traffic that actually needs it. The catch is the architecture only works as well as that probability is honest, and the backing lab found two silent ways it stops being honest.

2. What the tier has to guarantee, and the limits it can’t negotiate

Host: Okay, before we get to the two silent failures, I want to pin down what this tier actually has to promise. What does the backing lab say it’s on the hook for?

Guest: Four things. It has to be cheaper and faster than the LLM without getting worse on the answers it keeps, it has to escalate the rest at a known, agreed error rate, it has to detect when the probability stops meaning what it meant when you set the threshold, and the wording of the question itself has to be reviewed and versioned like any other config, not just a string someone typed once.

Host: That last one feels easy to skip past. So what can’t you negotiate on to hit those four — where does the lab say people get burned trying to cut corners?

Guest: Three hard constraints. The threshold is fitted, not chosen — it’s a function of the model, the wording, the option count, and the traffic, so changing any one of those silently changes your error rate without an alarm going off. Calibration has to be measured on your own labelled traffic, because vendor accuracy numbers, even the most cited one, are labelled by LLM consensus, so they measure agreement with other LLMs, not correctness. And option count is a hard ceiling, not a dial — in their tests three options supported a threshold that held up on held-out data, seventy-seven didn’t support one at all.

3. Rules, model, LLM: the request flow and the monitor that keeps it honest

Host: So walk me through what actually happens to a request once it hits the system. You said rules, then the decision model, then escalation — what’s that path look like in practice?

Guest: Every request goes to the cheapest tier that can honestly decide it. Rules grab the certain cases first, no probability needed. What’s left goes to the decision model, but it only keeps an answer if its top probability clears that fitted threshold — anything below gets punted to an LLM or a human, and those escalated outcomes get fed back in as labels too.

Host: And that feedback loop is the re-fitting monitor you mentioned — it’s not a nice-to-have bolted on afterward, it’s load-bearing.

Guest: Exactly, because the threshold was fitted on a snapshot of traffic, and traffic drifts. Without the monitor watching whether stated confidence still matches actual accuracy, you’ve just got a number that was correct once and nobody’s checking if it still is.

4. Three ways the probability lies without raising an alarm

Host: So walk me through what actually broke in the lab, because ‘the monitor wasn’t watching’ is still pretty abstract. Give me the first concrete failure.

Guest: Take the question ‘Is this product review positive?’ On a thousand labelled reviews it scored 52.9% — basically a coin flip, the majority baseline — and 998 of those answers were stated at 100% confidence. Just rephrase it to ‘Does the reviewer like the product?’ and the same rows jump to 91.2%. A tier built on the first wording keeps every answer because every answer clears threshold, nothing escalates, nothing looks broken — it’s just quietly wrong.

Host: That’s one wording landmine. What’s the second failure, the one with the 77 options?

Guest: At 77 intent options, answers stated at 96% confidence were right only 64% of the time, while answers stated at 9% were right 23% — the signal is miscalibrated in both directions at once, so no single threshold gets error below 10%, and one temperature fix can’t repair it since it moves every probability the same way. The fix is structural: split into a coarse choice then a fine one within the group. And the third failure is a vendor swap behind an API-compatible endpoint — same URL pattern, different model’s probabilities — so your fitted threshold is now operating on numbers it was never calibrated against, and a falling escalation rate in that moment could mean the system got better or went blind, and nothing short of labelled samples of what it kept tells you which.

5. The trade-off curve: coverage, error, and what it actually costs

Host: So once you’ve split a huge option set into coarse-then-fine, how do you actually pick where the threshold sits? Is there a correct number you’re aiming for?

Guest: There isn’t one correct number, there’s a curve: on three-way financial sentiment our fitted threshold kept 61% of traffic at 11% error on held-out data, and pushing it higher keeps less traffic but errs less — which side of that line you want is a business call about what a wrong kept answer costs you versus what an escalation costs, not a default anyone can hand you. That’s also why the split pays for itself: one 77-way question is one call and a probability that means nothing, while coarse-then-fine is two calls and two probabilities that each have a real chance of being calibrated, and the extra call is cheap compared to a threshold you can’t fit at all. But remember the tier only saves money on the 61% it actually keeps — the other 39% still pay for both the decision model and the LLM, so the decision call has to stay cheap, and as you scale, what strains isn’t compute, since a median 44 ms call barely moves, it’s keeping the labelled outcomes fresh enough for the monitor to trust that 61% number.

Host: So the fleet is the easy part and the labelling budget is the thing you actually have to plan for before you commit to a coverage target.

6. When the decision model is a guardrail, it’s also an attack surface

Host: Okay, before we leave the budget conversation entirely, I want to flag something that changes the math again: when this decision tier is acting as a guardrail, isn’t it also just… a target? Like, if the probability is the thing standing between a harmful request and approval, someone’s going to try to game that probability.

Guest: Exactly right, and it’s worth naming explicitly: a decision model used as a guardrail is an attack surface with a probability attached. Crafted inputs that push a harmful request just over the threshold are the decision-tier equivalent of prompt injection, and the only way you fail closed is if the tier escalates rather than approves — which means guardrail deployments want a deliberately high threshold and a slow path that’s genuinely independent. And the question wording itself is configuration an attacker would love to quietly edit, so it needs the exact same change control as the policy it’s encoding.

7. Watching it and shipping it: the observability signals and the deployment checklist

Host: So given that an attacker or just ordinary drift can shift the threshold’s meaning without tripping anything, what actually needs to be on a dashboard for this to be trustworthy day to day? Walk me through what you’re watching once it’s live.

Guest: Every single decision gets logged with its probability, which wording version produced it, and which model revision scored it — that triple is non-negotiable because without it you can’t even ask when something changed. Then you track escalation rate over time and alert on movement in either direction, since a drop looks like success but can mean the model quietly got overconfident. Alongside that you keep pulling labelled samples continuously, not just at launch, and bin stated confidence against actual accuracy — ECE gives you the headline number, but the bins are where you catch a one-sided or inverted miscalibration hiding behind a fine average. And you watch realised error on the answers you kept, measured against the exact target the threshold was fitted for, because that’s the number that tells you whether the guarantee is still true.

Host: And that logging discipline is really what turns wording changes, threshold updates, and model swaps into reviewed events instead of silent config edits — the same care you’d give the policy itself. So if someone’s building this tomorrow, that’s the checklist: evaluate the wording against alternatives, fit the threshold on one labelled set and check it on a held-out one, write down the target error and the cost of escalation and own it, keep sampling kept answers forever, re-fit on a schedule and on any change, and treat even an API-compatible model swap as a re-fit rather than a shrug. Get that right and the probability actually means what it claims to — which was the whole point. Thanks for walking through all of this.

Guest: Thanks for having me — and that last point is the one worth remembering above all: a calibrated decision tier isn’t a model you deploy once, it’s a claim you keep re-earning every time anything underneath it moves.

Not covered

The planner wanted these and found nothing in the source to support them:

A live walkthrough of actually fitting a threshold on a specific company’s traffic
A head-to-head cost comparison across Jev, Laya, Decisions API, and Clef pricing for this architecture specifically

Generated from this page by Claude Sonnet 5 on 2026-10-10, spoken by Kokoro-82M running locally. Two synthetic voices, not a recorded conversation. Every claim is drawn from this page — where it differs from the text above, the text is correct.

Problem

Much of what a production AI system sends to a model is not generation. It is a bounded question — which queue, which tool, is this in scope, is this safe — paid for at generation’s latency and price, and returned as text that has to be parsed back into the answer. A decision model answers the bounded question directly, from a set you define, with a probability over the answers.

That probability is the whole design. Act on the confident answers, escalate the rest, and the expensive model only sees the traffic that needs it. The architecture works exactly as well as the probability is honest — and the backing lab found two ways it is not, both silent.

Requirements

Answer bounded decisions faster and cheaper than an LLM, without becoming less correct on the answers it keeps.
Escalate the answers it should not keep, at a known and agreed error rate.
Detect when the probability stops meaning what it meant when the threshold was set.
Make the question’s wording a reviewed, versioned part of the system.

Constraints

The threshold is fitted, not chosen. It is a function of the model, the question’s wording, the number of options, and the traffic. Change any of them and the error rate the threshold delivers changes, without anything raising.

Calibration has to be measured on your data. Vendor accuracy figures come from vendor-chosen sets — the most cited one is labelled by LLM consensus, so it measures agreement with LLMs rather than correctness. A threshold needs labelled outcomes from your own traffic.

Option count is a hard limit, not a tuning knob. In the backing lab, three options supported a threshold that held on held-out data; seventy-seven did not support one at all.

Request Flow

flowchart TD
    R([Request]) --> RU{"Rules decide it?"}
    RU -->|yes| A1([Act: deterministic])
    RU -->|no| D["Decision model<br/>typed question, one pass"]
    D --> P{"Top probability ≥ threshold<br/>fitted on your labelled traffic"}
    P -->|yes| A2([Act on the decision])
    P -->|no| L["Escalate: LLM or person"]
    L --> A3([Act on the slower answer])
    A2 -. "sampled, labelled, logged" .-> M["Calibration monitor<br/>re-fit threshold on drift"]
    L -. "outcomes" .-> M
    M -. "new threshold" .-> P

A request goes to the cheapest tier that can decide it. Rules take what is certain. The decision model takes what is bounded, but only keeps answers whose top probability clears a threshold fitted on labelled traffic; everything below goes to an LLM or a person. Kept answers are sampled and labelled, escalated ones contribute their outcomes, and a monitor re-fits the threshold when the relationship between stated confidence and actual accuracy moves.

The monitor is not optional. Without it the threshold is a number that was right once.

Failure Modes

A phrasing that is wrong with total confidence

In the backing lab, “Is this product review positive?” scored 52.9% on 1,000 human-labelled reviews — the majority baseline — with 998 answers stated at 100%. “Does the reviewer like the product?” scored 91.2% on the same rows. A tier built on the first wording keeps every answer, because every answer clears any threshold, and escalates nothing. Nothing is malformed, nothing is slow, nothing raises. The only defence is to evaluate the wording on labelled data before it ships, and to treat a wording change as a deploy.

Confidence that inverts at high option counts

At 77 options the lab measured answers stated at 96% being right 64% of the time, while answers stated at 9% were right 23%. A threshold on that signal keeps confidently wrong answers and escalates some right ones. A single temperature cannot repair miscalibration that runs both ways — fitting one left ECE unchanged. Split the decision instead: a coarse choice first, then a fine one within the chosen group, so no single question carries dozens of options.

A threshold carried across a model swap

Decision APIs are compatible by design — one vendor’s client works against another’s endpoint by changing a URL. The probabilities that come back are a different model’s. A threshold fitted on the old model delivers an unknown error rate on the new one, and the lab’s synthetic tests show the shape: a threshold fitted on calibrated records misses its target on overconfident ones.

Escalation that never happens

The failure above, seen from the escalation side: if the escalation rate drops to near zero, the tier has either become very good or stopped being able to tell. Without labelled samples of the kept answers, the two look identical.

Scaling

The decision model is cheap per call — the backing lab measured a median of 44 ms per 3-option question and 203 ms per 77-option one on a laptop GPU — and several questions about the same state share one pass. What scales badly is not throughput but labelling: the monitor needs fresh labelled outcomes in proportion to how fast traffic drifts, and those cost human time. Size the labelling budget before sizing the fleet.

Security

A decision model used as a guardrail is an attack surface with a probability attached. Inputs crafted to push a harmful request above the threshold are the decision-tier version of prompt injection, and they fail closed only if the tier escalates — so the guardrail case wants a high threshold and a slow path that is genuinely independent. The question wording is configuration an attacker would like to change; it belongs under the same change control as the policy it encodes.

Trade-offs

Coverage against error

Every threshold trades the share of traffic the tier keeps against how often it is wrong on what it keeps. On three-way financial sentiment the lab’s fitted threshold kept 61% of traffic at 11.0% error on held-out data; raising it keeps less and errs less. The right point is a business decision about what a wrong kept answer costs against what an escalation costs — not a default.

One question with many options, or several with few

One 77-way question is one call and an unusable probability. A coarse-then-fine split is two calls and two probabilities that each have a chance of meaning something. The extra call is cheap; a threshold that cannot be fitted is not.

Hosted against open weights

A hosted decision API needs no GPU and no checkpoint, and its probabilities change when the vendor ships a new version. Open weights pin the model — the lab’s results are tied to one revision — and make the calibration yours to measure and keep.

Cost

The tier saves money only on the traffic it keeps. At 61% coverage, 39% of requests pay for the decision model and the LLM — so the decision call has to be cheap enough that 0.61 × (LLM cost) of saving outweighs 1.0 × (decision cost) plus the labelling the monitor needs. At low coverage the tier is a cost, not a saving. Compute it from your own coverage curve, not from a vendor’s latency table.

Observability

Every decision with its probability, its wording version, and its model revision.
Escalation rate over time, with alerts on movement in either direction — down is as suspicious as up.
Reliability on labelled samples, binned: stated confidence against actual accuracy. ECE is the summary; the bins are where a one-sided or inverted miscalibration shows.
Realised error on kept answers, against the target the threshold was fitted for.

Production Deployment

Before a decision tier keeps its first answer

The question’s wording was evaluated on labelled data, and alternative wordings were compared.
Option counts are small enough to calibrate; large label sets are split coarse-then-fine.
The threshold was fitted on one labelled sample and checked on a held-out one.
The target error rate and the cost of an escalation are written down and owned.
Kept answers are sampled for labelling continuously, not only at launch.
The threshold is re-fitted on a schedule and on any model, wording, or traffic change.
A model swap — including to an API-compatible vendor — is treated as a re-fit, not a config change.

What the backing lab does not cover

Fine-tuned models, score questions, and hosted decision APIs. Laya’s card says fine-tuning drives its headline accuracy, so the lab’s numbers describe the out-of-the-box model. The architecture’s rules — fit, hold out, monitor — do not depend on which of these you use; the numbers do.

Hands-on Lab

Laya 0.4.1 at a pinned revision, zero-shot, on three human-labelled sets: 5,000 decisions, committed as numbers. It measures accuracy against baselines, calibration error with fits judged on held-out data, reliability by bin, the coverage-error curve, and whether a threshold fitted for a target error holds — and it found one rewording that moves a yes/no question from chance to 91%. Read the lab documentation →

labs/decision-tier-calibrationproduction-shaped

Interview Questions

Your decision model scores 94% on the vendor benchmark. Can you skip the LLM for answers above 0.9?

Not on that evidence. The benchmark’s labels may not be ground truth — the most cited one is LLM consensus — and accuracy says nothing about whether 0.9 means 90%. Sample my own traffic, label it, check calibration bin by bin, fit the threshold for the error rate I can tolerate on one half, confirm it on the other, and monitor it afterwards.

The tier's escalation rate dropped to almost zero after a prompt change. Good news?

Probably not. A wording change can make a decision model confidently wrong — in the backing lab one phrasing scored chance with every answer at 100%. A falling escalation rate is consistent with a model that improved and with one that lost the ability to doubt itself, and only labelled samples of the kept answers tell them apart. Roll back the wording until they do.

You need to route among 80 support intents. Would you use one decision-model question?

No. At high option counts the probability stops being a usable escalation signal — the lab measured 96% stated confidence being right 64% of the time at 77 options, with no threshold reaching 10% error. Route coarse-then-fine: a handful of groups first, then the intent within the group, each a question small enough to calibrate.

Why not just apply temperature scaling and move on?

Because a single temperature moves every probability the same way, and real miscalibration is not always one-sided. In the lab it improved one case genuinely, made another look better only by turning a broken model into an honest coin flip, and left the rest neutral or worse. Measure the reliability bins before and after on held-out data; keep it where it helps.