LLM Routing: The Cascade Architecture Eating Every AI Stack

Somewhere in the last year, "which model should we use?" quietly became a wrong question, the way "which server should run our app?" became a wrong question when load balancers arrived. The right question is now "which model should see this request?", answered fresh, per request, in milliseconds. That's LLM routing, and the specific version winning everywhere is the cascade: cheap decider first, expensive thinker only when needed.

This page is the architecture deep dive. (If you want routing applied to a domain, the support-triage and email pieces are cascades wearing work clothes.)

The cascade in one diagram's worth of words

Every request hits a fast, cheap decision model first, Jev being the current loudest example. Two things happen there: the model either handles the request outright (when the task is itself decision-shaped: classify, route, score, verify) or triages it (rules on which downstream model, if any, deserves it). Attached to every verdict is a confidence signal. Above threshold, the verdict stands and the pipeline moves on. Below it, the request escalates: to a mid-tier model, a frontier model, or a human, depending on stakes.

The canonical receipt is the fraud-detection build: Jev judged 100 emails in 1.42 seconds, only the uncertain slice went to Kimi K3, and the hybrid scored 96 out of 100 for about $0.07 total (numbers as reported by the author). Read that as a general law, not a fraud fact: near-frontier accuracy at roughly a hundredth of frontier cost, because the expensive model never sees easy cases. Easy cases are most cases. That asymmetry is the entire business case.

Why the router should be a decision model

You can route with rules (brittle), with embeddings (fuzzy), or with a small chat LLM (workable, but you're parsing prose to extract a routing decision, which is a bug farm). A decision model is the routing layer with the costume removed: the question "which tier should handle this?" is literally a constrained-choice question, the exact format these models exist for. Milliseconds matter here too; a router that adds a second of latency taxes every request in the system, including the 90 percent that were easy. Reported decision latencies in the directory (sub-second verdicts, 10-per-second sustained in game loops) are what make the router effectively free.

The four design decisions that matter

Where the confidence threshold sits. This is your cost-versus-accuracy dial, and it should be set per route, empirically: run a few hundred labeled cases, plot accuracy against escalation rate, pick the knee. Guessing the threshold is how cascades get a bad name.

What happens below threshold. Escalation targets are policy: billing questions might cascade cheap → frontier → human, while "delete my account" should skip straight to whatever your lawyers said. Irreversible actions never ride a single unescalated verdict, at any confidence.

Feedback capture. Every escalated case is a labeled example of your cheap tier's edge. Reviewed weekly, those cases tell you whether the fix is a sharper question (usually; see judge-question craft) or a genuinely hard category that should stay escalated forever.

Version pinning. The router's questions and thresholds are production config; change them like you'd change an API contract, with versions and a diff, because a silent router tweak shifts cost and accuracy everywhere downstream at once.

Routing is one lever. The full cost playbook, cascades included and in payback order, is reduce LLM costs.

Frequently asked questions

What is LLM routing?

Deciding, per request, which model (or human) should handle it, usually via a fast cheap model that either resolves the request itself or escalates it with a confidence signal. The dominant pattern is the cascade: cheap first, expensive only for the uncertain slice.

How much does a cascade actually save?

The reported reference case hit 96/100 accuracy at about one percent of frontier-only cost, because the expensive model saw only hard cases. Savings scale with how easy your median request is, which for most products is "very."

Doesn't the router add latency?

A decision-model router adds milliseconds, and for the majority of requests it removes the frontier round-trip entirely, so median latency usually drops. A chat-model router is where the latency tax shows up.

When is a cascade the wrong choice?

When every request genuinely needs frontier reasoning (rare), when volume is too low for cost to matter, or when you can't yet label a few hundred cases to set thresholds honestly. Build the eval set first; the evals guide is the prerequisite reading.

Is this the same as "model routing" startups sell?

Same family; the difference is owning the routing questions yourself versus renting a generic router. Your routing criteria encode your product's policies, which is exactly the part worth keeping in-house.