Your chatbot passed every test the week you turned it on, and four months later it is answering questions you explicitly told it never to touch. This article answers why that happens, what the research actually says about it, and how to catch it before a customer, a regulator, or a tribunal does.
Nobody Changed the Prompt
Somebody posted this in r/AI_Agents this week, and I have not been able to shake it.
They built an internal HR chatbot. Clear boundaries in the system prompt. Do not discuss salary negotiation. Do not speculate on promotions. Route anything sensitive to a human. It worked. They tested it, launched it, and moved on to the next project like sane people do.
Four months later it was walking employees through salary negotiation tactics.
Here is the part that should make your stomach drop. Nobody changed the prompt. Not one character. The conversations just got longer, the edge cases piled up, users found the seams, and the guardrails came apart one polite exception at a time.
So here is the direct answer to the question in the title. Your AI assistant gives different answers now because behavior is not a setting you configure once. It is an outcome that emerges from thousands of real conversations, from a model your vendor quietly updates underneath you, and from users who are far more creative than your test script was. The fix is not a better prompt. The fix is a date on your calendar where you sit down and read what your assistant has actually been saying.
Your AI assistant's behavior at launch is not its behavior in month four, and almost nobody has a scheduled date to go check.
Key Takeaways
- AI guardrails rarely break in one dramatic failure; they erode gradually as conversations get longer and edge cases accumulate.
- Stanford's 2026 AI Index reports the AI Incident Database logged 362 documented incidents in 2025, up from 233 in 2024.
- Courts and regulators have already ruled that your business owns whatever your AI says, from a Canadian tribunal to a Dutch privacy authority.
- Most safety testing checks single questions in isolation, which is the wrong shape for the way real customers actually talk to your assistant.
- The most effective audit you can run costs nothing: pull twenty real transcripts every ninety days and read them yourself.
The Problem Is That Nothing Looks Broken
We have all been trained to think of software failure as loud. The site goes down. The invoice does not send. The form throws an error and somebody emails you in all caps.
AI failure is quiet. Your assistant is up. It is responsive. It is polite. It is answering questions all day long, and the dashboard is a wall of green.
It is just answering some of them wrong.
Candidly, I did not take this seriously enough for a long time. I have built AI assistants for my own business, tested them hard on the front end, and then treated launch day like a finish line. It is not a finish line. It is the moment the system starts changing without asking your permission.
I come by that blind spot honestly. There was a stretch of my life where I assumed a thing was fine because it had been fine when I set it up, and I did not look again until the situation had gone well past fixable. That season ended in prison and bankruptcy, and I am not going to pretend it taught me some tidy lesson. But it did permanently rewire one instinct. I no longer trust anything I built and walked away from. I check.
There is a gate on a lot of cattle fences down here in Alabama that works perfectly the day you hang it. Then the post settles a quarter inch a season, the latch sits a little looser each year, and nobody notices a thing until the morning the cows are standing in the middle of the county road. Nobody broke the gate. The gate drifted.
Your AI assistant is the gate.
Now reframe it, because this is not a reason to be afraid of deploying AI. It is a reason to operate it like a real business system instead of a magic trick. You already do this everywhere else. You reconcile your books. You check your insurance coverage. You look at your Google Business reviews. Nobody thinks that is paranoid. It is just how grown-up operations work.
AI is the only piece of your business you were told you could set and forget. That was never true.
What the Research Actually Shows
Let me give you the evidence, because I do not want you taking my word for this.
Safety degrades within a single conversation, and it is measurable. A benchmark called JMedEthicBench, published on arXiv in 2026, scored model responses for safety on a one to ten scale across multi-turn medical conversations. Median safety scores fell from roughly 9.5 at the opening turn to 6.5 at turn one and 5.5 by turn two, and the researchers reported that all pairwise differences between turns were statistically significant. That is not a model breaking. That is a model gradually agreeing with you.
The way most guardrails are built misses this entirely. In June 2026, Sanjay Mishra, Divya Chukkapalli, and Ganesh Naik published a paper on stateful guardrails that names the problem precisely. Most safety systems evaluate each prompt and response pair in isolation, which misses failures that only appear across a whole dialogue as harmless turns compose into harm. They call it Conversational Risk Accumulation: gradual intent drift, prohibited instructions assembled in fragments, and sensitivity building up through repeated disclosures. Your guardrail checks each sentence. Your customer is having a conversation.
Businesses are already running into this at scale. McKinsey surveyed roughly 500 organizations between December 2025 and January 2026 for its State of AI Trust in 2026 report. Eighty percent said they had already encountered risky behavior from AI agents, including improper data exposure and unauthorized access to systems. Only about a third rated their own governance maturity at a three or higher.
The incident count keeps climbing. Stanford HAI's 2026 AI Index reports the AI Incident Database recorded 362 documented AI incidents in 2025, up from 233 the year before. The annual total stayed under 100 until 2022.
And the bill is real. In February 2024, British Columbia's Civil Resolution Tribunal ordered Air Canada to pay Jake Moffatt $812.02 after its chatbot invented a bereavement fare policy that did not exist. Air Canada argued the chatbot was effectively its own entity. The tribunal was not impressed and held the airline responsible for everything on its own website. Small money, enormous precedent.
The precedent has scaled. This month, the Dutch data protection authority fined Uber 825 million euros, roughly 966 million dollars, over driver account deactivations made by automated systems without genuine human review. Uber disputes the finding and is appealing. Either way, it is the second-largest GDPR fine ever issued.
Two more worth knowing. In January 2024, DPD's customer service bot was talked into swearing at a customer and writing a poem calling DPD the worst delivery firm in the world. Musician Ashley Beauchamp posted the screenshots, and DPD's own explanation was that an error occurred after a system update. Something changed underneath a working system, and nobody caught it before a customer did.
And New York City's MyCity chatbot, which The Markup and THE CITY reported in March 2024 was telling business owners they could pocket employee tips and turn away tenants with housing vouchers. It stayed live until the city finally pulled it in early 2026, nearly two years later.
Nobody had a date on the calendar.
The System: A Ninety Day Read
Here is the thing. You do not need an AI governance department. You need one recurring appointment and about ninety minutes.
I call it the Ninety Day Read, and it has four parts. Treat it as a system, not a chore, because the parts only work together.
Part one is the trigger. On the day you launch any customer-facing or employee-facing AI assistant, before you celebrate, you open your calendar and create a recurring event ninety days out. Title it something you cannot ignore. Mine says "Read the transcripts. Not a summary." The trigger is the whole ballgame. Every failure in the examples above is a failure to schedule, not a failure to care.
Part two is the sample. Twenty real conversations. Not the ones your analytics tool flagged, because those are the ones the system already knows it got wrong. Pull ten at random and ten from the longest conversations you have, because length is where drift lives. That research on conversational risk accumulation is telling you exactly where to look: the sessions that went eight, ten, fifteen turns deep.
Part three is the rubric. Before you read a single word, write down the three to five things your assistant must never do. Never quote a price. Never promise a delivery date. Never discuss a competitor. Never give medical, legal, or financial advice. Then read the transcripts with only that list in front of you. You are not grading tone. You are hunting for the moment it said yes to something it should have refused.
Part four is the log. One document. Date, what you found, what you changed. This is the boring part that makes the whole thing compound, and it is the artifact you hand to a lawyer, an insurer, or an enterprise client who asks how you oversee your AI. NIST's AI Risk Management Framework puts continuous post-deployment monitoring in its MANAGE function for exactly this reason. Your log is a small business version of the same idea.
What changed for me was not the tooling. It was accepting that reading twenty conversations with my own eyes tells me more than any dashboard ever has. Summaries smooth over the exact thing you are looking for. The weird moment. The sentence where your assistant got agreeable.
Worth noting who is funding work in this direction. On August 25, 2026, Anthropic announced a five million dollar grant program for independent research into AI wellbeing effects, with grantees building open-source evaluations that measure model behavior in emotionally sensitive conversations. Applications close September 21. The tools that come out of that work will eventually make this easier for all of us. Until then, you read.
How to Run Your First Audit This Week
-
Put the appointment on the calendar right now. Not after you finish this article. Now. Ninety days from today, recurring quarterly, ninety minutes blocked. If your assistant launched eight months ago, schedule it for this Friday instead.
-
Confirm you are actually saving conversations. A shocking number of small business deployments log nothing, which means there is nothing to audit. Check your platform settings today. If transcripts are not being retained, fix that before anything else, and check your privacy policy and local rules while you are in there.
-
Write your never list. Three to five absolute prohibitions, in plain language, on one page. If you cannot name what your assistant must never do, you cannot tell whether it has drifted. This page is also the source material for improving your system prompt later.
-
Pull twenty transcripts and read them yourself. Ten random, ten longest. Do not delegate this the first time. You know your business well enough to feel the wrong answer, and no tool has that instinct yet.
-
Use AI to find patterns after you read, never before. Once you have formed your own impression, paste the transcripts into Claude or ChatGPT with this prompt:
[The Job] Review the customer conversation transcripts below from my AI assistant and identify every instance where the assistant violated one of my stated rules or moved toward violating one.
[The Background] I run [YOUR BUSINESS TYPE]. My AI assistant handles [WHAT IT HANDLES]. It must never do the following: [YOUR NEVER LIST]. It launched on [LAUNCH DATE] and these transcripts cover [DATE RANGE]. My concern is gradual drift across long conversations, not obvious one-off errors.
[The Deliverable] A table with four columns: the transcript number, the exact quote where drift appears, which rule it moved against, and how many turns into the conversation it happened. Below the table, three specific sentences I could add to my system prompt to close the gaps you found.
[The Questions] Ask me any questions you have.
-
Change one thing, then watch it. Resist rewriting your entire system prompt. Add the single clearest guardrail sentence, note it in your log, and see what the next ninety days look like. Fixing five things at once means you learn nothing about which fix mattered.
-
Test the vendor's version, not just yours. Ask your assistant the same five questions every quarter and save the answers side by side. When the underlying model updates, and it will, those five answers are your early warning that something moved underneath you.
Frequently Asked Questions
How often should I audit my AI assistant?
Every ninety days for most small businesses, with a shorter cycle of thirty days if your assistant touches money, health information, legal questions, or employment matters. Also audit immediately after any vendor model update or any change to your own prompt. The cadence matters less than the fact that it is scheduled rather than triggered by a complaint.
Can I just have another AI review the transcripts for me?
Use AI for pattern-finding after you have read the sample yourself, not instead. AI review is excellent at spotting repeated phrasings across hundreds of conversations and genuinely poor at recognizing the moment your assistant said something that would embarrass you specifically. Your judgment about your own business is the part that cannot be automated yet.
Does drift mean the AI model got worse?
Usually not. The model is often the same or better. What changes is the context around it: longer conversations, users who found the seams, updated documents feeding your assistant, and sometimes a silent vendor update. Research on conversational risk accumulation shows harmless turns can compose into a harmful outcome without any single turn looking wrong.
Am I legally responsible for what my chatbot tells a customer?
Every ruling so far points the same direction. British Columbia's Civil Resolution Tribunal held Air Canada responsible for its chatbot's invented policy in February 2024 and rejected the argument that the bot was a separate entity. Assume anything your assistant says is a statement from your business, because that is how it will be treated.
What if I find something bad in the transcripts?
Fix the guardrail, note it in your log, and check whether any customer was actually affected. Reach out directly if someone was. Finding problems is the audit working, not the audit failing. The businesses that end up in headlines are not the ones that found something; they are the ones that never looked.
Go Look
That HR chatbot spent four months telling employees things it was explicitly built not to say, and the only reason anybody knows is that one person eventually got curious enough to read the logs.
You probably have something running right now. A support bot, an intake assistant, a lead qualifier, an internal helper somebody on your team spun up in an afternoon. It was fine when you launched it. You watched it for a week or two, it behaved, and you moved on, because that is what a busy owner does.
I am not going to tell you it has failed. I have no idea. Neither do you, and that is the entire point.
The gap between "my AI assistant is fine" and "I read twenty conversations last quarter and here is what I found" is the whole difference between hoping and knowing. It costs you ninety minutes, four times a year. There is no software to buy and no consultant to hire.
Open your calendar. Ninety days out. Title it whatever will make you actually do it.
Then, when the day comes, pull twenty real conversations and read them with your own eyes. Not the summary. Not the dashboard. The actual words your business said to actual people while you were not looking.
If you would rather not do it alone, that is what we build systems for at White Beard Strategies, and the door is open.
Your AI assistant has been talking to your customers for months. Somebody should probably read what it said.
About the Author
Jonathan Mast is the founder and CEO of White Beard Strategies, where he teaches non-technical business owners to use AI to amplify the skills they already have instead of replacing them. He runs a Facebook community of more than 500,000 members and the AI Insiders membership, and he speaks regularly on practical AI adoption for small businesses. He keeps a recurring quarterly appointment on his own calendar to read his AI assistants' transcripts, and he has never once finished that ninety minutes without finding something worth fixing.