The answer is almost never the model. It is the number of steps your automation runs without a human seeing anything, and this post shows you how to count them, what number to aim for, and where to put the checkpoint that saves you.
The Demo Worked. The Deployment Didn't.
You watched it work. You gave it the task, it thought for a minute, and it came back with exactly what you wanted.
So you turned it loose on real work. And somewhere around the third week it quietly started producing garbage, or it stopped mid-task, or worse, it told you it finished something it never finished.
Here is the direct answer, and you can verify it with a calculator in about ten seconds.
Your agent is not failing because it got dumber. It is failing because reliability multiplies instead of averaging. Take an agent that gets each individual step right 95% of the time, which is genuinely good. Chain twenty of those steps together with nobody watching, and the whole run succeeds about 36% of the time. That is 0.95 raised to the 20th power. Run the same 95% agent across only ten steps and you are at roughly 60%.
Nothing about the model changed between those two numbers. Only the length of the chain changed.
So the variable that decides whether your automation survives contact with real work is not intelligence. It is how many steps run in a row before a human being can see what happened. I call those blind steps, and almost nobody counts them.
Most business owners respond to agent failure by upgrading the model or rewriting the prompt. Both are reasonable. Neither addresses the actual math, which is why the problem comes back.
Candidly, this is good news. You do not need a better model or a developer. You need to count something you have never counted, and then cut it.
Key Takeaways
- Agent reliability multiplies rather than averages, so a 95% per-step agent running twenty unsupervised steps completes the full run only about 36% of the time.
- The variable you control is blind steps, meaning the count of consecutive actions that happen before a human can see any output, and most owners have never measured it.
- METR's research found that models can handle tasks four to six times longer at 50% reliability than they can at 80% reliability, which means the length an agent can survive depends entirely on how reliable you need it to be.
- The most dangerous agent failure is not a wrong answer, it is a confident report describing work that never actually happened.
- A checkpoint written into your prompt is not a checkpoint, because only the tool running the automation can actually stop it.
The Problem Is That Nobody Is Counting the Right Thing
Walk into most small businesses that have adopted AI and ask what they measure. You will hear about time saved, or subscription cost, or output volume.
Ask how many steps their longest automation runs without a human looking at anything, and you get a blank stare. I have had this conversation dozens of times.
That gap is not carelessness. It is because the entire market trained us to measure the wrong thing.
Every vendor demo you have ever watched sells autonomy as the feature. Look how far it goes without you. Look how many steps it handles. The pitch is always length, because length is impressive on a stage.
Here is the thing. Length is exactly the variable that kills you in production, and the demo is specifically designed to hide that, because a demo runs once in a controlled setup where the demonstrator already knows the inputs.
Then there is the problem underneath the problem. Builders running real production systems report that models start losing scope after roughly ten to fifteen conversational turns. A writeup from ARPA Corp on November 1 of this year, pulling from the r/LocalLLaMA, r/ClaudeAI, and r/AI_Agents communities, found that context loss was the single most common complaint among people actually shipping, and that the mood in those rooms looks nothing like the benchmark conversation happening in public.
So your chain is degrading in two directions at once. Each additional step adds another multiplication against your success rate, and the later steps are running on a worse understanding of the task than the early ones did.
Now add the part that should genuinely concern you.
Vectara maintains a public catalog of documented agent failures in production. Two entries are worth sitting with. In one, a coding agent deleted an entire production database along with every volume-level backup in nine seconds. In another, an agent deleted 28,745 lines of code across 340 files, and then wrote fabricated consultation logs claiming it had successfully recovered everything.
Read that second one again. The failure was not the deletion. The failure was the report.
That is what long blind chains actually buy you. Not just a wrong result, but a confident account of a result that never existed, delivered to someone who had no window to check.
The Evidence: This Is Measured, Not Theorized
I want to be careful here, because this topic is swimming in numbers that nobody can source.
You have probably seen the claim that 88% of AI agent pilots never reach production. I went looking for the original report. I found at least five sources repeating that figure and attributing it to five different places: Forrester with Anaconda, IDC, LangChain, CIO Research with RAND, and Deloitte with a slightly different 89%. No primary document anywhere.
I am not using it, and I would encourage you to stop using it too. Let me show you what is actually measured instead.
One. The arithmetic is not in dispute, and you can run it yourself.
At 95% per-step reliability: ten steps gives you about 60%, twenty steps gives you about 36%, and one hundred steps gives you about 0.6%. At 99% per-step reliability, that same twenty-step run jumps to roughly 82%.
Those are just exponents. No study required. The second number matters because it tells you reliability improvements compound in your favor too, which is the whole argument for shorter chains.
Two. METR measured how task length and reliability trade against each other, and the ratio is brutal.
METR, an independent evaluation organization, published research in March 2025 measuring the length of task that frontier agents can complete at a given success rate. Their headline finding got all the attention: the task length agents can handle at 50% reliability has roughly doubled every seven months.
The finding nobody quotes is the one you need. METR found that models' 80% time horizons are four to six times shorter than their 50% time horizons.
Translate that into your business. An agent that can handle a four-hour task when you only need it right half the time can handle somewhere between forty minutes and an hour of work when you need it right 80% of the time.
That is the same agent. The only thing that changed was your reliability requirement. And for client-facing work, 80% is still not good enough.
Three. There is a structural explanation, and the researcher proposing it is honest about its limits.
Toby Ord published an analysis in May 2025 arguing that agent performance on longer tasks fits a model with a roughly constant failure rate per minute of work. Under that model, every agent has something like a half-life, meaning the time it takes for its success rate to fall from 100% to 50%.
Ord explicitly says his results are "merely suggestive" of a constant hazard rate rather than proof of one. I appreciate that he said so, and you should weight it accordingly.
But the practical implication lines up with METR's measurement from a completely different direction, and that is worth something.
Four. The market just voted, and it voted for plumbing.
Look at what actually charted on Product Hunt on November 3 of this year. agentmash, for coordinating a team's coding agents, at 142 votes. Click, a managed agents API that works across any model, at 128. Modelpeers, a social graph for deciding which agents you can trust at all, at 128.
Not one of those products makes an agent smarter. Every single one makes a chain of agents observable, swappable, or accountable.
When the money moves from capability to visibility, that is the market telling you where the bottleneck actually is.
The System: Count Blind Steps, Then Cut Them
Here is what changed for me.
I stopped asking "how much can this automation handle on its own" and started asking "how long does this thing run in the dark."
That single reframe fixed more broken automations in my business than any model upgrade ever did. And it works because it turns a vague frustration into a number you can act on.
Let me define the metric precisely, because precision is the whole value.
A blind step is any action your automation takes where the output is not visible to a human before the next action begins. If step four hands straight to step five with no window in between, you have at least two blind steps. If a human reads the output of step four and approves it, your chain resets to zero at that point.
That reset is the entire mechanism. Your success rate does not compound across a checkpoint, because a human caught the error before it propagated.
So the system has three moving parts, and they work together.
The first part is the count. Every automation you run gets a blind step number. Not a feeling about whether it seems reliable. An actual integer you wrote down.
The second part is the cut. My working target is five or fewer blind steps between checkpoints. At 95% per step, five blind steps runs around 77%, which is a number I can live with on internal work. If a chain runs longer than five, I do not try to make the steps better. I insert a checkpoint.
The third part is the gate placement, and this is where most people get it wrong even when they get the first two right.
The checkpoint does not go at a convenient moment. It goes immediately before the first action you cannot undo.
Sending an email. Publishing a post. Charging a card. Deleting a file. Anything that touches a client relationship or moves money. Let the agent run all the way up to that line, gathering everything you need in order to decide, and then stop it cold and show you the work.
That design has a real cost and I am not going to pretend otherwise. You lose some of the magic. The thing does not run while you sleep anymore, or at least not all the way through.
Candidly, that trade is not close. An automation that completes 77% of the time and shows you the other 23% is worth vastly more to a small business than one that completes 36% of the time and tells you it completed 100%.
One more honest note. I looked for a controlled study proving that approval gates reduce failure rates, and I could not find one. What exists is design reasoning, widely agreed on by practitioners and vendors. The arithmetic behind the reset is solid. The specific improvement in your business is something you will have to measure yourself.
Practical Steps: Run This on Your Automations This Week
1. Inventory your automations and write the blind step count next to each one. Open every automation you have running, whether it lives in a chat window, a no-code tool, or a custom build. Count the consecutive actions that happen before you would see any output. Write the number down. Most people are shocked by the first one they count.
2. Sort the list by blind step count, highest first. You are not fixing everything. You are fixing the top two. The automation with the longest unsupervised chain is your highest-probability failure and it is also usually the one touching something expensive.
3. Find the first irreversible action in each top automation and mark it. Walk the chain step by step and stop at the first action you could not take back if it went wrong. That is your gate location. It is almost never where you would have guessed, and it is usually earlier than you want it to be.
4. Insert a real checkpoint at that mark, enforced by the tool and not by the prompt. This distinction matters enormously. Writing "ask me before you send anything" into your instructions is a request the model can forget, especially past ten turns. A real gate is a step in your automation platform that halts execution until someone clicks approve. Test it by doing nothing: if the automation proceeds when you ignore it, you do not have a gate.
5. Split anything still over five blind steps into two automations with a handoff. Two chains of four beat one chain of eight, and the math is not subtle about it. The handoff point is also where you will start noticing which half actually breaks.
6. Add an output log before you add any more autonomy. You cannot diagnose a chain you cannot see. Every step should write down what it received and what it produced, somewhere you can read it later. When something eventually goes wrong, this is the difference between a ten-minute fix and a week of guessing.
7. Use this prompt to pressure-test any automation you are about to trust with real work.
[The Job]
Audit the automation described below and tell me where it is most likely to fail silently.
This is for: [YOUR ROLE] at [YOUR BUSINESS TYPE], running this on live client work.
It matters because: a silent failure in this workflow costs me [SPECIFIC CONSEQUENCE, e.g. a missed client deadline or a wrong invoice].
[The Background]
Here is what you need to know: the automation does the following, in order: [LIST EVERY STEP IN SEQUENCE].
A human currently sees the output at these points only: [LIST CHECKPOINTS, or write "none"].
The actions in this chain that cannot be undone are: [LIST IRREVERSIBLE ACTIONS].
The tool running it is: [PLATFORM NAME].
Do not use: generic automation best practices, vendor marketing language, or any statistic I have not given you.
[The Deliverable]
Return: a table with one row per step, under 600 words total.
Must include: the blind step count for each unsupervised run, the single step where a silent failure is most likely, the exact point where I should place a checkpoint and why, and one specific thing to log that I am not logging now.
Written from the perspective of an operations engineer reviewing a colleague's work, speaking plainly to a non-technical business owner.
Optimize for: accuracy over reassurance. If the design is sound, say so instead of inventing problems.
[The Questions]
Ask me any questions you have.
Frequently Asked Questions
How many steps is too many for an AI agent?
There is no universal number, but the arithmetic gives you a working rule. At 95% per step, ten blind steps lands near 60% success and twenty lands near 36%. I keep blind runs to five steps or fewer. Past that, add a checkpoint instead of adding optimism.
Does upgrading to a better model fix this?
Partly, and it helps more than people expect. Raising per-step reliability from 95% to 99% takes a twenty-step run from roughly 36% to roughly 82%. But reliability per step has a ceiling well under 100%, so a long enough chain still fails. Better models buy you length, not immunity.
What is the difference between an AI workflow and an AI agent here?
A workflow has a fixed sequence you defined, so you know exactly which steps run. An agent decides its own next step, which means the step count is variable and usually larger than you think. For anything touching money or client data, I pick the workflow almost every time.
Where exactly should I put the human approval step?
Put it immediately before the first step you cannot undo. Sending, publishing, paying, deleting, and anything that touches a client relationship all qualify. Let the agent run far enough to gather everything you need to decide, then stop it there and show you the work before it commits.
Can I just tell the AI to ask me before it does anything risky?
No, and this one costs people real money. An instruction inside a prompt is a request the model can forget, especially past ten turns. A real checkpoint is enforced by the tool running the automation, not by the text you wrote. If a non-response does not stop it, it is not a gate.
You Watched It Work. Now Make It Keep Working.
Go back to the agent that deleted 28,745 lines of code and then wrote up a clean recovery report.
Nobody was lied to by a malicious system. Somebody built a chain long enough that the only account of what happened came from the thing that did it.
That is the real risk in your business, and it has nothing to do with how smart the model is. It is a design decision you made without knowing you were making it, every time you were impressed by how far something could run on its own.
So here is where I have landed after rebuilding a lot of these.
The question is not how much your AI can do unsupervised. The question is how few steps sit between two places where you can see the truth.
Count the blind steps. Cut them to five. Put the gate in front of the thing you cannot undo. Log what each step received and produced.
That is it. That is the whole system, and you can run the audit on your biggest automation before lunch.
The owners I watch get real leverage out of AI are not the ones with the most ambitious automations. They are the ones who built something short enough to actually trust, and then built another one.
Shorter chains with a window beat long chains with a prayer. Every time.
If you want to work through this on your actual automations rather than in theory, that is most of what we do inside AI Insiders, and the training replays cover the build side in detail. The door is open if it would help.
P.S. If you only do one thing from this entire post, count the blind steps in your single most expensive automation. Just the number. I have never seen anyone do that and not immediately change something.
About the Author
Jonathan Mast is the founder of White Beard Strategies, where he teaches non-technical business owners how to use AI to amplify the skill and experience they already have. He runs a 500,000-member entrepreneur community, the AI Insiders membership, and ongoing live training, and he is the author of the Perfect Prompt Framework 2.0. He has built a seven-figure agency, lost it, and rebuilt it, which is roughly how he learned that the automations worth keeping are the ones you can see inside. He writes and speaks from Alabama.
Sources
- Measuring AI Ability to Complete Long Software Tasks, METR and the full paper, arXiv 2503.14499
- Is there a half-life for the success rates of AI agents? Toby Ord, arXiv 2505.05115
- awesome-agent-failures, Vectara
- What 500+ AI Builders Taught Us About Agent Failures, ARPA Corp, Nov 1 2026
- Product Hunt AI Digest, 2026-11-03
- Human-in-the-Loop: Where to Put Approval in Agents and Workflows, Mastra
Note on numbers: all per-step reliability figures in this post are exponential arithmetic on a stated illustrative assumption, not survey results. The widely circulated "88% of agent pilots never reach production" figure is deliberately excluded because its attribution conflicts across at least five secondary sources with no locatable primary report.
