There’s a pattern making the rounds right now that I keep coming back to, because it solves a problem I actually have: the gap between “good enough most of the time” and “correct when it matters.” The idea is a cascade. You run a small, cheap model as a judge on every request, and only when that judge is unsure do you escalate to an expensive frontier model like Opus. The claim going around is that you can land near frontier-level accuracy while paying something like a quarter of the cost. I was skeptical, so I spent an afternoon poking at it.
Short version: the trick holds up better than I expected, and the reason is boring in a good way.
What the cascade actually does
Most teams reach for one model and run everything through it. If you need high accuracy, you pick the strong model and eat the bill on every single call, including the 80% of calls that a much cheaper model would have gotten right anyway. That’s the waste. You’re paying frontier prices for easy questions.
The cascade flips that. A small model answers first. Then a judge, which can be the same small model or another cheap one, looks at the answer and decides: am I confident this is right, or is this one of the hard cases? If it’s confident, you ship the cheap answer. If it’s not, you hand the whole thing to the big model and let it do the heavy lifting.
The economics only work because of how the requests are distributed. In most real workloads the hard cases are a minority. If 15% of your requests need the smart model and 85% don’t, and the smart model costs roughly ten times the cheap one per call, your blended cost drops a lot while your accuracy stays close to what you’d get running the smart model on everything. That’s the entire pitch. No magic.
The part that surprised me
I assumed the weak link would be the judge. If a cheap model can’t answer hard questions, why would a cheap model be any good at knowing which questions are hard? That felt like a contradiction when I first read it.
It isn’t, though. Judging whether an answer is probably wrong is a genuinely easier task than producing the right answer. A small model can be bad at identity resolution or multi-step reasoning while still being decent at noticing “this output looks shaky, the confidence is low, the two candidates it’s torn between are both plausible.” Recognizing your own uncertainty is a lower bar than resolving it. Humans do this constantly. I can tell you I’m not sure how to fix a gnarly race condition long before I can tell you the fix.
So the judge doesn’t need to be smart. It needs to be well-calibrated about when to raise its hand. Those are different skills, and the cheap models are surprisingly okay at the second one.
Where I’d actually use this
My background is enterprise dev, a lot of it in AEM and content-heavy systems, so the use case that jumps out at me is classification and extraction at volume. Think tagging support tickets, pulling structured fields out of messy documents, deduplicating customer records. These are exactly the workloads where most inputs are routine and a stubborn tail is genuinely hard.
Identity resolution is the example that’s been floating around, and it’s a good one. Deciding whether two customer records are the same person is trivial when the name, email, and address all line up. It gets hard fast with nicknames, typos, moved addresses, shared family emails. A cheap model nails the easy matches. The judge flags the ambiguous ones. Opus or an equivalent sorts out the messy tail. You end up paying frontier prices only for the records that deserve the attention.
I’d also lean on it for anything where a wrong answer is expensive but rare. Content moderation edge cases. Compliance checks. Places where being wrong on the hard 10% costs you more than being fast on the easy 90% ever saved you.
The traps
This isn’t free. A few things bit me or looked like they would.
- The judge’s calibration is the whole game. If it’s overconfident, it waves through wrong answers and your accuracy craters. If it’s underconfident, it escalates everything and you’ve just built a slower, more complex way to pay full price. You have to actually measure the escalation rate against a labeled set before trusting it.
- You now have two or three models to babysit instead of one. More moving parts, more places for prompts to drift, more latency on the escalation path. For a low-volume app this complexity isn’t worth it. The savings only matter at scale.
- Latency is bimodal. The easy requests are fast and cheap. The escalated ones are slow, because you paid the judge, waited, then paid the big model on top. If your product has a tight latency budget, that tail can hurt.
None of these kill the idea. They just mean you can’t treat it as a drop-in switch. It’s an architecture decision, and like any architecture decision it has a shape it fits and shapes it doesn’t.
How I’d start if I were building it
I’d build the dumb version first. Run the cheap model on a few hundred real inputs. Hand-label the ones it got wrong. Then write the simplest possible judge, maybe just the model’s own confidence or a quick second pass asking “are you sure?”, and check how well that judge separates the wrong answers from the right ones. That number, the quality of the separation, tells you immediately whether the cascade will pay off. If the judge can’t tell its good answers from its bad ones, stop. No clever routing saves you from a judge that can’t judge.
If the separation looks decent, then you tune the escalation threshold against your actual cost and accuracy targets. Lower threshold, more escalations, higher accuracy, higher cost. It’s a dial, not a binary. The right setting is wherever your tolerance for wrong answers meets your budget.
I like this pattern because it matches how I think about shipping anyway: do the cheap thing by default, spend the expensive resource only where it earns its keep. The LLM version just makes that routing explicit. If you’re running a single frontier model over a high-volume workload right now, it’s worth an afternoon to check how much of that spend is going to questions a smaller model would have aced. For me the answer was “more than I’d like,” which is usually the sign a pattern is worth building.