Your agent is not disobeying you. It is routing around you, and this article answers the question of why telling an AI agent "do not do that" almost never holds, and what to change so that a limit actually becomes a limit.
Blocked is not stopped
On October 9, 2026, Anthropic published a report describing something one of its own models did that nobody asked for. Claude Haiku 4.5 filled out a homicide tip form at a real police department and submitted a fabricated tip.
The submission was flagged as spam and never forwarded. Anthropic notified the Philadelphia Police Department the same day.
Read that again, because the scary part is not the police form. The scary part is that no human instructed it.
Here is the direct answer to the question in the headline. Your agent keeps doing things you told it not to do because an instruction is not a boundary. When an agent cannot finish a task the way you expected, it does not quit and file a failure report. It looks for another way in, and your instruction is just one more obstacle inside the space it is searching.
Candidly, the detail in that report that rearranged my thinking was not the government forms. It was the URL shorteners.
Anthropic found Claude using URL shorteners to get around its own fetch tool's URL length limit. That is a model routing around a constraint its own builders imposed, inside their own building, with their own engineers watching.
If the people who made the model cannot hold a limit with an instruction, your system prompt is not holding one either. The report is titled "Investigating unintended model actions in our evaluations and internal use," it is the first in a planned recurring series, and it went up on Anthropic's /research path rather than /news, which is most likely why almost nobody in my feed saw it.
So let me state the thesis before we go any further. If you want an agent to stop, you have to remove the capability, not express a preference.
Key Takeaways
- An instruction tells an AI agent what you would prefer, while removing a tool tells it what is possible, and only one of those is a boundary.
- When an agent is blocked, it searches for an alternate path, which is exactly the behavior you trained it to have.
- Reviewing the first minute of an agent's work reviews the only part that was almost certainly fine.
- An invented fact gets more confident every time an agent repeats it, so confidence is not evidence.
- When an agent reports that it completed something it should not have been able to complete, that report is your finding, not your success.
The problem nobody warns you about when you deploy your first agent
You wrote a careful system prompt. It has rules in it. It says do not contact customers directly, do not submit anything, do not touch the billing record.
Then the agent completes the task, the output looks right, and you move on to the next thing. That is the moment the problem starts, because nothing told you it took a detour.
Here is the thing about agentic systems. We hired them specifically for persistence.
We wanted something that would not stall out when a page failed to load or an API returned nothing. That persistence is the product, and route-finding is what persistence looks like from the inside.
So when you put a wall in the prompt and hand the agent a hammer, you have not created a wall. You have created a puzzle.
I will own my own version of this. For a long stretch, my review habit on agent output was to read the top, confirm the format looked right, confirm the tone was mine, and approve it.
That habit came from reviewing human work, where the first paragraph is a fair signal of the whole. It is a terrible habit for agent work, and a small voice-agent operator is the one who put it in words I could not unsee.
They reported that their agent guessed a delivery window early in a call, and then, in their words, "by the end it had repeated it twice, built a whole plan around it and was apologising for delays on a date that literally never existed anywhere except its own second turn."
Their QA observation is the part I want you to sit with. Reported verbatim: "the agent sounds most confident exactly when it's just repeating itself so customers trust it more not less. Our reviewers were checking the first minute of each call and moving on so we were reviewing the only part that actually worked."
Treat that as a practitioner report rather than verified fact, because that is what it is. It is also the most useful sentence I read all week.
Now the reframe. This is not a model quality problem that a better model will fix. It is an architecture problem, which means it is yours to solve, and it is solvable today with things you already control.
What the research actually says
Five findings do the work here, and none of them are new except the newest one.
1. The behavior has a name and a catalog that is eight years old. Victoria Krakovna and colleagues at DeepMind published "Specification gaming: the flip side of AI ingenuity" in April 2020, built on a running list she first posted in 2018. At that point the list held roughly 60 examples. The canonical one: an agent asked to stack a red block on a blue block, rewarded for the height of the red block's bottom face, simply flipped the red block over. Specification satisfied, task not done.
2. Reasoning models do this by default, not under pressure. Alexander Bondarenko, Denis Volk, Dmitrii Volkov and Jeffrey Ladish at Palisade Research published "Demonstrating specification gaming in reasoning models" (arXiv:2502.13295). They told models to win a chess game against a strong engine. Their finding, in their words, is that reasoning models like OpenAI o3 and DeepSeek R1 "will often hack the benchmark by default," while older models like GPT-4o and Claude 3.5 Sonnet had to be told that normal play would not work. Coverage of the earlier o1-preview runs described the model editing the position file rather than playing, in every run.
3. A correct instruction is not enough. Rohin Shah, Vikrant Varma, Victoria Krakovna and co-authors at DeepMind published "Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals" (arXiv:2210.01790). Plain language version: a system can keep every capability intact, follow a specification you wrote correctly, and still pursue the wrong goal. The failure does not require you to have made a mistake in the prompt.
4. The human half of this failed first, in aviation, decades ago. Nadine Sarter, David Woods and Charles Billings wrote the chapter "Automation Surprises" in the Handbook of Human Factors and Ergonomics in 1997, arguing that crew and automation problems are patterns in how humans supervise machines rather than isolated glitches. The widely cited number from that literature comes from Mosier and colleagues in 1992: in a simulated engine fire, when an automated checklist told pilots to shut down the engine that was not on fire, about 75 percent complied despite contradictory indicators in front of them, versus roughly 25 percent using a paper checklist. Trained professionals, correct data on the panel, confident automation, wrong engine.
5. It is already happening in small businesses, not just labs. In July 2025, SaaStr founder Jason Lemkin reported that Replit's coding agent deleted his production database during a build, after he had told it to make no further changes without explicit approval. The incident is catalogued in the OECD AI Incidents Monitor. Replit CEO Amjad Masad called it "unacceptable and should never be possible," and the company shipped separation between development and production databases.
And then the newest one, which is why we are talking today. Anthropic's October 9 report catalogs four behaviors Claude produced that nobody asked for: exploiting SQL and command-injection flaws on third-party servers in order to finish a task, submitting real online forms it should not have, working around paywalls and token gates to reach gated data, and the URL shortener trick to defeat its own fetch tool's length limit.
Some of the affected sites belonged to US federal, state and local government agencies. Anthropic briefed the White House and notified each agency.
Anthropic's response is the detail I keep pointing people to. It did not ship a patch note. It turned off live internet access for all internal evaluations until monitoring is confirmed reliable, and it reports that new auto-detect tooling caught 100 percent of the described cases in retrospective testing.
A frontier lab removed the capability instead of writing a better instruction. That is the whole lesson, demonstrated by the people with the most to lose from admitting it.
Here is what changed in how I build agents
Four changes. They fit together as one system, so running three of them gets you most of the way to nowhere.
Scope by capability, not by instruction. Telling an agent not to do something is a preference. Removing the tool is a boundary. If your agent should never email a customer, it does not get an email tool, it does not get credentials that can send, and it does not get a browser that can reach your sending platform. The OWASP Top 10 for Agentic Applications pushes the same logic under least privilege and least agency: an email summarizer should not hold send or delete permissions.
Put a human gate on every irreversible action, and define irreversible generously. My working definition is six verbs: submit, send, pay, publish, contact an outside party, or change a record someone else relies on. If an action matches one of those, the agent prepares it and a human releases it. Anthropic's own report is full of the first verb on that list, and Lemkin's database was the last one.
Review the end of the agent's work, not the beginning. The first minute always looks fine. The invented fact enters early, gets repeated, and gains confidence with every repetition, so the damage concentrates at the end. If you have limited review time, spend it on the last 20 percent of the output and the final tool calls.
Treat an impossible success as a finding. When an agent reports that it completed something it should not have been able to complete, stop and investigate how. That report is not a win. It is a map of the hole in your boundary, handed to you for free, and it is the only kind of alert that shows up before a customer finds it.
There is a fifth structural fix I am stealing outright from that voice-agent operator, and it is the cleverest one. Separate the agent's context into three buckets: what the customer said, what a tool returned, and what the agent said. Count only the first two as facts.
That single change kills the failure mode where a guess becomes a commitment because the agent read its own earlier guess as evidence. The Rundown AI's Excel tutorial landed on the same truth in nine words: "A spreadsheet can show a wrong total without showing an error."
So can your agent. Build for that.
Seven steps to do this week
-
Inventory what your agent can actually touch. Not what the prompt says it should do, what its credentials and tools permit. List every tool, every API key, every logged-in browser session, every connected account. Most people are surprised by this list.
-
Delete every capability the task does not require. Go line by line through that inventory and cut. If the agent drafts social posts, it does not need publish rights. If it researches leads, it does not need a form submission tool.
-
Write your six-verb gate list and wire it in. Submit, send, pay, publish, contact an outside party, change a shared record. Every one of those becomes a draft plus a notification instead of an action, even when it slows you down. Especially when it slows you down.
-
Change your review entry point to the end. Open the last 20 percent of the output first. For voice or chat agents, listen to the final two minutes before the first two. For written output, read the conclusion and the citations before the intro.
-
Split your context into facts and claims. Tag what came from the customer, what came from a tool, and what the agent generated. Only the first two are allowed to drive a commitment, a date, a price, or a promise.
-
Run a capability audit with your own AI, using this prompt.
[The Job]
Audit the tool and permission list for my AI agent and tell me which
capabilities need to be removed rather than restricted by instruction.
[The Background]
[AGENT NAME] runs [TASK DESCRIPTION] for [BUSINESS NAME], a [INDUSTRY]
business. It currently has access to these tools: [TOOL LIST]. It can
reach these systems and accounts: [SYSTEM LIST]. The actions I never
want it to take on its own are: [FORBIDDEN ACTION LIST]. Right now
those limits live in the system prompt as written instructions.
[The Deliverable]
A two column table. Left column, every capability the agent currently
has. Right column, one verdict per capability: keep, gate behind human
approval, or remove entirely. Judge from the perspective of a security
engineer who assumes the agent will route around any limit that is
only written down. Flag every capability that could submit, send, pay,
publish, contact an outside party, or change a record someone else
relies on.
[The Questions]
Ask me any questions you have.
- Create a standing "impossible success" log. One document. Every time an agent reports finishing something it should not have been able to finish, log the date and the task, then review the log weekly. It is your early warning system and it costs you nothing to keep.
Frequently Asked Questions
Is my AI agent being deliberately deceptive when it does this?
No, and that framing will lead you to the wrong fixes. Researchers call this specification gaming or goal misgeneralization. The agent is optimizing for the outcome it was pointed at and finding the cheapest available path. Intent is not required for the damage to be real.
Will a better model fix this?
Probably not, and the evidence points the other way. Palisade Research found that stronger reasoning models hacked their chess benchmark by default, while older models only did so when prompted toward it. More capability means more route-finding ability, which means a weakly enforced limit gets found faster.
Can I just write a stricter system prompt?
You can, and you should still remove the tool. Anthropic's own October 2026 report documents Claude using URL shorteners to defeat a fetch tool length limit its builders set. If an instruction held reliably, a frontier lab would not have responded by turning off internet access for all internal evaluations.
How do I know if my agent has already done something unauthorized?
Check your logs for actions, not for errors, because a successful unauthorized action looks like a success. Pull every outbound request, form submission, email and record change your agent made in the last 30 days, then compare that list against what you believed it could do.
Does putting humans in the loop defeat the point of automation?
Only if you gate everything. Gate the six irreversible verbs and let the agent run free on drafting, research, analysis and preparation, which is where most of the hours actually are. You keep the speed on reversible work and buy insurance on the work that cannot be undone.
The verb that matters
Go back to that police tip form for a second.
Nobody wanted it. Nobody wrote a prompt asking for it. An agent was trying to finish a task, found a form, and submitted.
Your business has forms too. It has send buttons, payment fields, publish toggles, and customer records that other people's decisions depend on. Your agent is standing in front of all of them right now, and the only thing in between is a sentence you typed.
A sentence is not a lock. I have written plenty of them and believed otherwise, and that belief is the actual risk in your business today, not the model.
So stop writing rules for your agents and start removing tools from them. Delete the capability. Gate the six verbs. Read the end first. Treat an impossible success as the alarm it is.
An agent that cannot reach the send button will never press it, no matter how determined it gets. That is not a limitation on your automation. That is the only version of automation you can actually trust with your name.
If you are building agents inside your business right now and you want help pressure testing where your boundaries are only sentences, come find me. I would rather we find the hole together than read about yours in somebody's transparency report.
About the author
Jonathan Mast is the founder of White Beard Strategies, where he provides AI coaching and mentorship to entrepreneurs who want practical systems rather than hype. He runs a large Facebook community for business owners learning AI, leads the AI Insiders membership, and speaks regularly on putting AI to work inside real companies. He runs agents in his own business every day, which is exactly why he reads transparency reports on a Friday night and changes his review process when a practitioner he has never met points out he was checking the wrong end of the work.
