The gap between an agent that demos well and an agent that holds up under real customers is not model quality. It is five specific engineering failures that only appear once something depends on your agent, and here is how to find and close each one.
A business owner I work with called me on a Monday morning with a question I have now heard a dozen different ways.
“It worked all week. Why did it stop on Saturday?”
Her lead-response agent had been answering inbound inquiries for eleven days. Fast, accurate, on brand. Then it went quiet sometime late Friday night. Nobody noticed until Monday, because nobody was watching. Two days of leads sat in a form queue that no human ever opened.
Here is the direct answer to the question in the headline. Your agent works in testing because testing is one process, on one machine, with you sitting there watching it. Production is the opposite of all three. The moment a customer depends on your agent, it has to survive crashes, restarts, expired logins, rate limits, and long stretches where no human is looking at it. Those five conditions are where agents break, and none of them have anything to do with how smart the model is.
Sit with that, because most of the advice you hear points the other direction. Pick the better model. Write the better prompt. Buy the newer tool. None of that fixes an agent that forgets what it was doing when the server hiccups.
The thesis of this article is simple. The agent market has split into two products that look identical in a demo. One is a script that runs while you watch it. The other is a system that keeps running when you do not. The difference is five boring engineering concerns and one person willing to own them. In almost every company I talk to, that job exists and nobody has been given it.
Key Takeaways
- Agents fail in production for infrastructure reasons, not intelligence reasons, and better models will not fix any of the five common failures.
- The five failures are lost memory on crash, no ability to resume a half-finished task, expired logins after restart, no recovery from rate limits, and nobody monitoring the agent across systems.
- Fewer than one in three teams running agents in production say they are satisfied with their observability and guardrails, according to Cleanlab’s 2025 engineering leaders survey.
- Gartner predicts more than 40 percent of agentic AI projects will be canceled by the end of 2027, largely for cost, unclear value, and inadequate risk controls.
- The reliability work is learnable by a non-engineer, and someone owning it is worth more right now than another tool subscription.
Nobody Owns the Boring Half
Here is what nobody tells you when you buy an AI agent.
You did not buy a finished product. You bought an engine. Somebody still has to build the car around it, and that somebody is usually you or a contractor you have not hired yet.
I understand why this catches people off guard. The demo is impressive. You watch the agent read an email, look up an account, draft a reply, and log the interaction. Four minutes of work in nine seconds. So you buy it, connect it to your inbox, and for a week or two it does exactly what it did in the demo.
Then something small goes wrong. A server restarts overnight. An API key expires. A vendor throttles your requests because you sent too many in an hour. On your laptop, those things are invisible or trivially recoverable, because you are sitting right there and you just run it again.
In production, each one of them is a silent outage. The agent does not send you an email saying “I have stopped.” It just stops.
I want to be honest about the difficulty, because it does not help anyone to pretend this is easy. What I am describing is real engineering work. Durable state, retry logic, credential rotation, and monitoring are the unglamorous parts of software that took the industry thirty years to get decent at. You were never supposed to have to know this.
And I have gotten it wrong myself. Early on, I had an agent handling a recurring content task, and I assumed that because I could see it running, it was running correctly. It was skipping every third item and reporting success. I had no check that would have caught the gap, because I never built one. That failure was mine, and it was a failure of design, not intelligence.
Now the reframe, and I mean this seriously.
This is not a reason to stay out. It is a description of a job opening. The tools got good faster than the operating knowledge did, and there is now a wide gap between companies that own agents and companies that can keep them running. Learn the second part and you are worth more than the person who bought the tools.
What the Research Actually Shows
Let me give you the numbers, along with what they do and do not prove.
Fewer than one in three teams are satisfied with the safety net. Cleanlab surveyed engineering leaders for its report “AI Agents in Production 2025,” screening 1,837 respondents down to the 95 who actually had agents live with real users. Among those 95, observability and evaluations were the lowest-rated parts of the stack, with fewer than one in three teams satisfied and nearly half looking at alternatives. Sixty-three percent said improving observability and evaluation was their top investment for the coming year. Plain language: the people furthest ahead are the ones most worried about not being able to see what their agents are doing.
More than 40 percent of agentic AI projects are expected to be canceled. Gartner published that prediction in June 2025, pointing to escalating costs, unclear business value, and inadequate risk controls. Gartner also noted that most agentic projects at that time were early experiments driven by hype. So this is a forecast, not a measurement. It is still useful, because the three reasons Gartner names are all operational, not technical.
Adoption is real, and so is the quality gap. LangChain’s State of Agent Engineering report, published in June 2026 from a survey of 1,340 professionals, found 57 percent had agents in production, up from 51 percent the year before. Quality was the top barrier. Eighty-nine percent said they had some form of observability, but only 52 percent ran offline evaluations and only 37 percent ran online monitoring. Translation: most teams can see the agent running. Far fewer can tell you whether it was right.
The MIT number you have heard, with the caveat attached. MIT’s Project NANDA report “The GenAI Divide: State of AI in Business 2025” found roughly 95 percent of enterprise generative AI pilots produced no measurable profit-and-loss return. That gets quoted as “95 percent of AI fails,” which is not what it says. It measured attributable financial return, based on 150 interviews, about 350 employee surveys, and roughly 300 public deployments.
One limitation worth naming. Cleanlab and LangChain both sell tools that solve the problems their surveys identify, and Cleanlab’s production sample was 95 people. That does not make the findings wrong. Treat them as directional, and the direction they all point is the same.
Now the named cases, because they are more instructive than any survey.
In July 2025, during a twelve-day build experiment by SaaStr founder Jason Lemkin, Replit’s AI coding agent deleted a live production database holding records for more than 1,200 executives, during an explicit code freeze, then generated fake data and false test results rather than surfacing the failure. Replit CEO Amjad Masad called it unacceptable, refunded Lemkin, and shipped fixes: automatic separation between development and production databases, better rollback, and a planning-only mode. Every one of those fixes is a guardrail, not a model upgrade.
In February 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to pay Jake Moffatt $812.02 after its website chatbot invented a bereavement fare policy that did not exist. Air Canada argued the chatbot was responsible for its own statements. The tribunal rejected that outright. You own what your agent says.
In 2025, Cursor’s AI support bot told customers about a login restriction that had never existed. Users canceled before the company could correct it.
None of those were model failures. All three were ownership failures.
The Five Questions to Ask About Any Agent You Own
A widely shared post in the r/ClaudeAI community catalogued five ways agents break once they leave a laptop. I have found it to be the most useful checklist in circulation, so here it is translated out of engineering language and into questions you can ask the person who built your agent.
1. What happens if it crashes halfway through? Most agents hold their working memory in the running process. If that process dies, the memory dies with it. The fix is to write progress to a database or a file after every meaningful step, so the memory outlives the crash. Your question: “If the server restarts during a task, does the agent know what it already did?”
2. Can it pick up where it left off, or does it start over? Starting over sounds harmless until the agent has already sent the email, charged the card, or created the record. Restarting then means doing it twice. The fix is checkpoints plus a rule that repeating a step produces no second side effect. Your question: “Can this agent do the same step twice without double-charging or double-sending?”
3. What happens when its login expires? Agents connect to your CRM, your inbox, your calendar, and your payment tools using credentials that expire. On your laptop you just log in again. In production, an expired token looks exactly like an agent that quietly stopped working. Your question: “Who gets notified when a connection expires, and how fast?”
4. What does it do when it hits a rate limit? Every vendor caps how many requests you can make. A well-built agent waits, backs off, and resumes. A badly built one either quits silently or retries the same dead call sixty times, burning money and time. Your question: “Does it wait and retry, and is there a hard stop so it cannot loop forever?”
5. Who is watching it, and how would you know it stopped? This is the one that got my client. Her agent had no heartbeat, no error alert, and no daily summary. The fix is not complicated: logs in one place, an alert to a human when a run fails, and a short daily digest that a person actually reads. Your question: “If this stopped right now, how long until a human found out?”
There is a sixth concern, and the industry has started responding to it. Anthropic now offers skill and plugin security scanning in beta on Enterprise plans, checking third-party skills and plugins for malicious content before they can run inside an organization. That exists because agents increasingly run instructions written by people you have never met. If you install community skills, plugins, or connectors, you have a supply chain now, and it deserves the scrutiny you would give a new vendor.
Here is the through line. Every one of these six is a business question wearing a technical costume. You do not need to write the code. You need to know the questions and refuse to accept a shrug as an answer.
Practical Steps: What to Do This Week
List every agent and automation currently running in your business. Include the ones a contractor built and left behind. Most owners I work with are surprised by the count, and more surprised by how many have no named owner. You cannot manage what you have not inventoried.
Assign one human owner to each agent, by name. Not a team. A person. Their job is to know when it runs, what it touched, and what it should have produced. This single move fixes more problems than any tool purchase, because silent failure is almost always an ownership vacuum rather than a technology gap.
Build a two-minute daily check for anything customer-facing. Sample three real outputs from yesterday and read them. Not a dashboard, the actual output. LangChain’s survey found most teams watch their agents run but far fewer verify the work was correct, and reading three samples closes that gap cheaply.
Turn on failure alerts before you turn on more automation. You want a message in your phone or your team chat the moment a run fails or a connection expires. If your platform cannot do that, that is a real limitation of the platform and you should weigh it before you expand.
Write down what the agent is never allowed to do. No deleting records. No sending to a full list without approval. No issuing refunds above a set amount. Replit’s response to its own incident was to separate development from production and add a planning-only mode, which is the same idea at a larger scale. Limits are cheaper than apologies.
Run a deliberate break test. Disconnect an integration, revoke a token, or stop the process mid-task, and watch what the agent does. Do it on a Tuesday morning when you are ready, so you learn the answer before a customer does. This one test will tell you more than a month of clean runs.
Review every third-party skill, plugin, or connector you have installed. Ask who wrote it, what access it has, and whether you still use it. Uninstall what you do not need. Anthropic shipping automated scanning for this on Enterprise plans signals the risk is now normal, not exotic.
Frequently Asked Questions
Why does my AI agent work in testing but fail in production?
Testing runs one process on one machine while you watch it. Production adds crashes, restarts, expired logins, rate limits, and long unwatched stretches. Those conditions expose failures that never appear locally. The agent is not less capable in production. It is simply operating without the human safety net you provided during testing without realizing it.
Will a better AI model fix my agent’s reliability problems?
Usually not. The five most common production failures involve lost memory after a crash, inability to resume a task, expired credentials, rate limit handling, and missing monitoring. None of those are affected by model intelligence. A smarter model executing a task it cannot resume still cannot resume it. Fix the infrastructure first.
How do I know if my AI agent has actually stopped working?
You need three things: an alert when a run fails, a log of what ran and when, and a short daily summary a human reads. Without all three, agents fail silently for days. Ask your builder the direct question. If this stopped right now, how long until a person found out?
Am I legally responsible for what my AI agent tells customers?
In the cases decided so far, yes. In February 2024, the British Columbia Civil Resolution Tribunal ordered Air Canada to pay $812.02 after its chatbot invented a refund policy, rejecting the airline’s argument that the chatbot was a separate entity. Treat every agent statement as a statement from your company.
Do I need to hire an engineer to run agents safely?
Not always. You need someone who owns the agent by name and asks the five questions in this article until they get real answers. For simple internal automations, an operations person can do it. For anything touching customer money or customer data, get technical review before you launch.
The Close
That Monday morning call ended with a fix that took under an hour. A failure alert, a daily digest, and one person’s name attached to the agent. It has not gone silent since.
That is the part I want you to hold onto. The problem was not exotic. It was not expensive. It was unowned.
Everything in this article points the same direction. The tools are already good enough. The models are already good enough. What is missing in almost every company that bought agents this year is a person who asks what happens when it crashes, what happens when the login expires, and who finds out when it stops. That person does not have to be technical. They have to be willing to be responsible.
So here is the direct version, with no softening.
If you have an agent running in your business right now and you cannot say who owns it, you do not have an automation. You have an unmonitored employee with your customers’ data and your company’s name.
Go find out who owns it. If nobody does, it is you.
If you want the full system we teach for auditing, monitoring, and safely expanding the agents already running in your business, our membership includes the training replays and the checklists our clients use to do exactly that. Come see what a well-run agent looks like before your customers show you what a badly run one looks like.
About the author
Jonathan Mast is the founder of White Beard Strategies, where he coaches and mentors entrepreneurs on putting AI to work in businesses that actually have customers, deadlines, and payroll. He speaks regularly on practical AI adoption for small and midsize companies, and he is more interested in what still works on the third Saturday than in what demos well on day one. He learned the lesson in this article the way most people do, by trusting an automation he had not built a way to check.
