The answer is the same architecture for both problems: put a small, cheap model in front of your expensive one to decide what deserves attention, and put a second small, cheap model behind it to check whether the decision it made actually follows your written rules. This article walks through how that two-stage setup works, what it costs, and how to build it this week.
Your AI Bill Is Mostly Paying for Nothing, and Your AI Agent Is Quietly Guessing
A practitioner published a write-up this week on building proactive AI agents that do not bankrupt you. His opening observation stopped me cold.
"In a typical inbox, 95% of incoming traffic is noise. Newsletters, automated order confirmations, LinkedIn updates, spam, and notification pings arrive all day."
Then he named the consequence. "If you invoke that loop on every newsletter and receipt, you end up spending tens or hundreds of dollars a month just to have a frontier model tell you to ignore an automated receipt."
If you have ever opened an AI usage dashboard and felt your stomach drop, that is the mechanism. You are not paying for intelligence. You are paying a very expensive model to read your Costco receipt.
Here is the direct answer. The fix is two stages, not one. Stage one is a cheap, fast model whose only job is deciding whether a human or the good model needs to see this at all. Stage two is a separate cheap model that reads your written policy, the facts, and the decision your agent just made, then answers one question: does this decision actually follow from the rules? If no, the request goes to a human.
Both stages use the same insight. Checking is a smaller job than doing.
And candidly, the second stage matters more, because the first one only costs you money. Someone on a finance team described the second failure mode in a sentence that should make any business owner sit up.
"A common way these agents go wrong is that they read a policy, read a request, and then confidently make a call the policy doesn't actually allow. The tools all worked fine. The reasoning slipped."
No error. No crash. Every tool returned a 200. The approval just went out, and nobody knew.
That is the thesis of this entire article: a cheap filter in front saves your budget, and a cheap checker behind saves your business.
Key Takeaways
- A cheap classifier in front of your expensive model handles the noise, so the expensive model only sees work worth paying for.
- A separate cheap checker behind your agent reads your policy and the agent's decision, then routes it to a human when the decision does not follow the rules.
- Stanford's FrugalGPT matched GPT-4 quality at up to 98% lower cost using a cheapest-first cascade.
- Verification is a far smaller job than generation, which is why a tiny model can usefully check a frontier model's work.
- You cannot check a decision against a rule that only exists in your head, so writing the policy down comes first.
The Problem: You Are Paying Frontier Prices to Triage Garbage, and Nobody Is Checking the Output
The two failures feel completely different from the inside.
The first is the one you notice. Your bill goes up, and you cannot explain why.
You set up an AI assistant to watch your inbox, your support queue, your lead forms. It works. You are delighted. Then the invoice arrives and the delight evaporates.
The reason is arithmetic nobody warns you about. Every item hitting that queue gets the full treatment: the whole reasoning loop, the whole context window, the whole premium model. The newsletter from a plugin vendor gets the same expensive consideration as the email from your biggest client.
You are paying a specialist consulting rate to have someone sort your junk mail.
The second failure is the one you do not notice, and it is worse.
Your AI agent reads your refund policy. It reads a customer's request. It approves the refund. The policy did not actually allow that refund, and the agent was confident anyway.
Nothing broke. There is no error log to check. The reasoning just slipped somewhere between reading the rule and applying it, and the output looked exactly as polished as the correct output would have looked.
I want to be honest about how hard this second one is, because the usual advice is useless. People tell you to "keep a human in the loop," which sounds responsible right up until you realize it means you personally reviewing four hundred decisions a week. That is not a loop. That is a job you just gave yourself.
Here is the thing. You do not need a human reviewing everything. You need something cheap enough to review everything, and smart enough to flag the handful that need you.
Which reframes the whole problem. Stop asking how to make one AI smarter and more trustworthy. Start asking how to arrange two or three cheap pieces so the expensive one only gets the work it deserves, and never acts alone on anything hard to undo.
That is not an AI question. That is an org chart question. You already know how to solve those.
The Evidence: Researchers, a Fintech, and a Hospital Trial All Landed in the Same Place
This is not a clever trick somebody invented on a podcast last month. Four separate communities arrived at it independently, and the research backs it up.
Stanford put a number on the savings in 2023. Lingjiao Chen, Matei Zaharia, and James Zou published FrugalGPT, which opens by noting that fees across popular model APIs "can differ by two orders of magnitude." Their system calls models in sequence, cheapest first, stopping as soon as one produces a satisfactory answer. The reported result: FrugalGPT matched the best single model with "up to 98% cost reduction," or beat GPT-4's accuracy by 4% at the same cost. Those are the authors' own numbers, and savings are task-dependent. The direction is unmistakable.
Berkeley confirmed it with a router instead of a cascade. The team behind RouteLLM at LMSYS and Berkeley's Sky Computing Lab trained small routers to decide, per query, whether a cheap model or GPT-4 should answer. They reported "cost reductions of over 85% on MT Bench, 45% on MMLU, and 35% on GSM8K as compared to using only GPT-4, while still achieving 95% of GPT-4's performance." The detail business owners should notice: their best router needed GPT-4 for only 14% of queries to hit that quality mark. The rest the cheap model handled fine.
A real fintech runs it at scale right now. Ramp built an internal router and opened it as a product. On its own product page, Ramp states Router "cut our LLM costs by 30% while making our features smarter and faster," attributed to CTO Rahul Sengottuvelu, with reported volume above 2.75 trillion tokens routed monthly at roughly 30 milliseconds of added latency. Those are Ramp's figures for Ramp's product, so weigh them accordingly.
On the checking side, a cheap judge is a credible judge. In Judging LLM-as-a-Judge, presented at NeurIPS 2023, Lianmin Zheng and colleagues found strong model judges reached "over 80% agreement" with human preferences, which they note is "the same level of agreement between humans." Meta shipped the product version, Llama Guard, a seven-billion-parameter safeguard model that reads a written policy taxonomy and classifies both the incoming request and the outgoing response against it. The policy is an input you edit, not code you rewrite.
And the oldest version of this idea has a catch. In March 2026, Johanna Westbrook and colleagues at Macquarie University published a randomised controlled simulation trial in BMJ Quality and Safety: 82 nurses, two hospitals, 1,160 dose administrations with embedded errors. Independent double-checking caught 77.7% of errors versus 66.3% for a single checker. But the benefit went almost entirely to experienced nurses, who improved 26%. Early-career nurses showed no significant gain, and the researchers found evidence of social loafing: people checked less carefully knowing somebody else was also checking.
Translate that into plain business language. A second set of eyes works when the checker is genuinely competent and genuinely independent. It fails when the checker is weak, or when both parties are quietly relying on the other one to catch it.
Which is precisely why you build the checker as a separate model with a separate prompt and one narrow question, instead of asking your agent to double-check itself.
The Solution: A Receptionist in Front, a Second Set of Eyes Behind
Forget the engineering vocabulary for a minute. You already run this system with people.
Stage one is your receptionist. Their job is not to solve the client's problem. Their job is to decide whether this call goes to voicemail, to a form, or straight through to the specialist. You pay a receptionist a fraction of what you pay the specialist, and the business works because of that gap, not in spite of it.
Stage two is the colleague who reads the contract before it goes out. They are not writing the contract. They are answering one question: does this match what we agreed to do?
That is the whole architecture. Cheap triage in front, cheap verification behind, expensive capability in the middle doing only the work that needs it.
Here is what makes this a right-now opportunity instead of a theory: the pieces got absurdly cheap this month, and four communities shipped them at once.
Microsoft released Microsoft-Decision-1, a model that does not generate text at all. It reads your options and returns calibrated probabilities across them in a single pass, post-trained from Qwen3.5-9B. Listed price is $0.042 per million input tokens, output free. Microsoft's own framing of why speed matters is the clearest explanation of latency I have seen: 100 milliseconds added to each of twenty sequential decisions adds two full seconds to a workflow. Every benchmark on Microsoft's listing is a vendor claim, so treat them as marketing until somebody independent tests them.
On the same Product Hunt board the same day, an 11-megabyte local model called ej launched doing the same classification job entirely on a phone. An open four-billion-parameter entrant showed up in r/LocalLLaMA that day too. Those are community reports, not verified performance.
A free, MIT-licensed Claude Code tool picked up 222 upvotes for the routing version, using Haiku 5.5 to pick the model and effort level per task. The builder's stated reason is the most relatable sentence in this article: "I kept forgetting to switch models, so I ran everything on high setting."
Notion published the enterprise version. It breaks AI tasks into categories and routes each by quality, latency, and cost. For auto-filling project tracker fields, high volume but low reasoning complexity, it uses "specialized, cost-efficient, fine-tuned models that cut latency in half and improve output quality." Notion also runs an internal LLM-as-a-judge system staffed by people it calls AI Data Specialists.
A router in front. A judge behind. Same two stages.
The part nobody can do for you is the policy. Your checker needs something to check against, written in plain language, in a file. Not in your head. In a file.
That requirement is the real gift here, and it has nothing to do with AI. Writing down "we never issue a refund past 60 days without my approval" forces you to actually decide it.
Practical Steps: Build This in Seven Moves
-
Count what percentage of your noisiest queue is actually noise. Pick one workflow where AI touches every item: inbox, support tickets, lead forms, invoices. Spend twenty minutes labeling a hundred recent items as noise or real. That ratio is the only number that justifies the build.
-
Write the policy down before you touch any tool. Write the rules in plain sentences a new hire could follow: dollar thresholds, time windows, who approves what, which categories are automatic no. This step is where most people quit.
-
Build stage one as a classifier with exactly one output. The filter should not summarize, advise, or draft anything. It returns a label: ignore, handle automatically, escalate to the good model, or escalate to a human. One job keeps it cheap and auditable.
-
Price both stages before you commit. Compare the per-item cost of your filter against a full agent turn on your expensive model. The practitioner whose write-up opened this article claims his classifier stage runs at roughly one hundredth of a full agent turn. That is his number, nobody verified it, and your ratio will differ. Run your own.
-
Build stage two as a separate model asking one question. Before the agent does anything hard to undo, a small separate model reads the policy, the facts, and the agent's decision, and answers only this: does this decision follow from the rules? If no, it goes to a human. Separate prompt, separate call, no self-grading.
-
Use this prompt to draft your policy file. Paste it into Claude or ChatGPT and fill in the brackets.
[The Job]
Help me turn my informal decision rules into a written policy file that a small AI checker model can use to verify whether a decision follows the rules.[The Background]
My business is [BUSINESS TYPE]. The workflow I am automating is [WORKFLOW]. The decisions the agent will make are [DECISION TYPES]. My current informal rules are [YOUR RULES IN WHATEVER MESSY FORM THEY EXIST]. The actions here that are hard to undo are [IRREVERSIBLE ACTIONS]. The person who handles exceptions today is [NAME OR ROLE].[The Deliverable]
A plain-language policy document, written from the perspective of an operations lead who has to defend every rule to an auditor. Each rule stated as one testable sentence with explicit thresholds and no ambiguous words like "reasonable" or "significant." Group the rules into automatic yes, automatic no, and must escalate to a human. Flag every place my rules contradict each other or leave a gap.[The Questions]
Ask me any questions you have. -
Run it in shadow mode for two weeks before you let it act. Let both stages log their decisions without executing anything, then review the log yourself. You are checking how much the filter would have saved and whether the checker flagged the right things. Only then turn the automation on.
Frequently Asked Questions
What is AI model routing, in plain English?
Model routing means a small, cheap AI model looks at each incoming task first and decides which model should handle it. Simple tasks go to a cheap model, hard tasks go to an expensive one. Berkeley's RouteLLM research found that routing sent only 14% of queries to GPT-4 while retaining 95% of its performance.
How much can a two-stage AI setup actually save me?
It depends entirely on how much of your traffic is noise. Stanford's FrugalGPT reported up to 98% cost reduction in its own experiments, and Ramp states its internal router cut company AI costs by 30%. Both are self-reported. Count the noise percentage in your own queue before believing any figure.
Can a cheap AI model really be trusted to check an expensive one's work?
Often, yes, because checking is a much smaller job than producing. NeurIPS 2023 research on LLM-as-a-judge found strong model judges agreed with human evaluators over 80% of the time, matching human-to-human agreement. The catch is that the checker must be a separate call with its own narrow question, never the same model grading itself.
Do I need a developer to build this?
Not necessarily. Building can mean configuring a tool somebody else made. Zapier and Make both support model selection and human approval checkpoints in their visual builders, and gateway products like Ramp Router offer routing behind a single endpoint. What you cannot outsource is writing your policy down in plain language.
What should I check first, cost or correctness?
Correctness. A high bill is annoying; a wrongly approved refund or a wrongly sent email costs you trust you may not get back. Build the checker behind your agent first, on the one workflow with irreversible actions, then add the cheap filter in front once the guardrail is proven.
The Close: Delegate the Checking Before You Delegate the Work
Go back to that practitioner's sentence. Ninety-five percent of what hits your inbox is noise, and you have been paying frontier prices to sort it.
But the sentence that should change what you do this week is the other one. "The tools all worked fine. The reasoning slipped."
That is what an unchecked AI failure looks like. Not a crash. Not an alert. A confident, well-written, completely unauthorized decision that nobody catches for three weeks.
So here is the move, as directly as I know how to say it.
Build it in two stages. A cheap filter in front so your expensive model only sees what matters. A cheap separate check behind so nothing irreversible happens without something reading your actual rules first. Write the rules down today, in plain sentences.
Then notice what you just learned about your human team.
This architecture works because verification is a much smaller job than the original work. That is equally true of people. Delegation feels impossible because we hand off the producing before we hand off the checking, when the checking is the smaller, more teachable half.
Delegate the checking first. Then delegate the work. That order was available the whole time, and a nine-billion-parameter model just proved it.
If you want to build this alongside people already running it, come join us.
Your AI does not need to be smarter. It needs a receptionist and a second set of eyes.
About the Author
Jonathan Mast is the founder of White Beard Strategies, where he provides AI coaching and mentorship to entrepreneurs and small business owners who want practical systems rather than hype. He runs a large Facebook community for business owners learning AI and leads the AI Insiders membership, and he speaks regularly on putting AI to work inside real companies. He is unreasonably fond of architectures where the cheap part does the boring work, which is probably why he spent a decade building systems before he ever built a course.
