Yes, and sooner than you think. This article explains why the open web is closing to AI agents, what replaced it, and how to move one workflow onto data nobody can revoke.
A builder posted on Reddit this weekend that sites which used to work had started blocking his AI agent. It was checking car listings. Cars.com began flagging it as a bot, and commenters reported the same thing happening on job boards.
That is how this usually goes, and the reason it is worth writing about is the shape of the failure. Not an error. Not a warning. A workflow that runs, reports success, and comes back with nothing in it.
The direct answer to the question in the title is yes, your AI workflow will break if it depends on reading a site you do not control, and the failure will be silent rather than loud. But the useful version of that answer is not a warning, it is a redirect. Because while borrowed access was being fenced off, sanctioned access to your own systems was being commoditized. Both of those things happened in the same eighteen months and they are the same story.
The web did not close. It got priced. And the businesses that come out ahead are not the ones with the cleverest workaround. They are the ones whose AI runs on data they already own.
Key Takeaways
- As of June 2026, more than half of all internet traffic is non-human, and 52 percent of crawler requests are for AI training, up from 22 percent in spring 2025.
- From 15 September 2026, Cloudflare began offering new domains one of two preset crawler configurations at onboarding, with the restrictive preset applied to sites that earn money from advertising.
- Federal computer crime law got weaker for site owners in August 2026, which means contracts, bot management, and licensing are now the real gate, not the courts.
- At the same moment, the sanctioned rails got built: more than 10,000 published MCP servers, agentic payment protocols, and cryptographic bot authentication heading into the IETF.
- Every one of those rails requires identity, permission, and usually a contract. The advantage goes to whoever owns the data, not whoever scrapes best.
The Problem: You Built on Ground You Do Not Own
Most small businesses got into AI workflows the same way. You found something tedious, you pointed an AI at it, and it worked. Competitor pricing. A supplier catalog. A directory listing. Job boards. Review pages.
It worked because the open web operated on an implicit exchange for twenty-five years: crawlers took content, and search sent back traffic. That exchange broke, and it broke fast.
MIT's Data Provenance Initiative published the first large longitudinal audit of this in July 2024, tracking robots.txt and terms of service across 14,000 web domains from 2016 to 2024. In a single year, 2023 to 2024, restrictions rendered roughly 5 percent of all tokens in the C4 training corpus, and 28 percent of its most actively maintained sources, fully restricted. Among the most heavily used news sources in that sample, nearly 45 percent of tokens went from unrestricted to fully restricted, up from 3 percent the previous year. Domains with no robots.txt at all fell from 20 percent in 2016 to near zero.
Then the infrastructure layer moved. Cloudflare became the first major provider to block AI crawlers by default for new domains on 1 July 2025. Fourteen months later, on 15 September 2026, it started offering every newly onboarded domain one of two preset crawler configurations, chosen based on whether the site earns money from advertising. Ad-supported sites get the restrictive preset: search allowed, training disallowed, agent crawling restricted on monetized pages. Sites that do not run ads get everything allowed.
Be precise about what that does and does not mean, because this gets exaggerated constantly. It is a preset offered at setup, not a silent lockdown, and the owner can change any of it during onboarding or later. The training setting works by publishing a preference in robots.txt rather than hard blocking. And part of the same change actually loosened things, migrating some existing customers from a hard block down to a published preference.
The direction still matters more than the mechanism. A default that used to be "take whatever you want" is now a question the site owner gets asked on day one.
I want to be fair about the source. That is Cloudflare's own telemetry from Cloudflare's own network, published by a company that sells the blocking product. It is primary and first-party, and it is also self-interested. Say both things.
But the numbers in its 2026 bot traffic reporting are hard to dismiss. More than 50 percent of internet traffic is now non-human. Fifty-two percent of crawler requests are for AI training as of June 2026, against 22 percent in spring 2025. And the line that should matter most to a small business: some of the most heavily crawled categories have seen human traffic decline as much as 40 percent in under a year, across retail, software, IT, and finance. Not just news.
And what if you thought the courts would sort this out? In August 2026 the Ninth Circuit vacated a preliminary injunction in Amazon v. Perplexity, holding that when a user directs an AI assistant to act on a website, it is the user and not the assistant's maker who accesses the site, on the reasoning that the assistant is a tool rather than a person and runs on the user's own device. Verify that holding against the opinion before you rely on it, because this area is moving fast and the practical point does not depend on the case. Federal computer crime law got weaker as a lever for site owners at the exact moment network-level blocking got stronger. That is why the enforcement moved to infrastructure. Blocking is now the tool that works.
The Evidence That the Traffic Is Not Coming Back
Here is the part that changes what your website is for.
Pew Research Center studied browsing data from 900 US adults in March 2025, covering 68,879 unique Google searches. When a Google AI summary appeared, users clicked a traditional search result in 8 percent of visits, against 15 percent when no summary appeared. They clicked a link inside the summary in 1 percent of visits. They ended the browsing session on 26 percent of pages with a summary, against 16 percent without.
I want to be careful here, because this is where a lot of commentary overreaches. The Reuters Institute Digital News Report 2026, published 16 June 2026 across 48 markets, found that 42 percent of people who use AI chatbots for news say they always or often click through to the original source. That sits between social media at 36 percent and search at 44 percent. So chatbots are not a black hole for referrals in the way the doom version of this story claims, and if you have read otherwise, check the number.
What the Pew data shows is narrower and more specific: the presence of an AI summary on a results page suppresses clicks on that page. That is the effect that should concern you, and it is measured behavior rather than self-report.
Cloudflare measured the same collapse from the other end, comparing crawl requests to referrals in June 2025. The ratios ranged from Anthropic's 70,900 to 1 down to Mistral's 0.1 to 1. Cloudflare notes honestly that native app traffic does not send a referrer header, so the figures may overstate the ratios by an unclear amount.
Even Wikipedia is feeling it. The Wikimedia Foundation disclosed in October 2025 that after updating its bot detection logic and reclassifying several months of data, human pageviews were down roughly 8 percent year over year. Notably, much of the traffic spike they had originally counted as human turned out to be bots built to evade detection.
I put all of that together and reach one conclusion. If the thing that now recommends your business does not send people to your site, your site's job has changed, and the data your AI reads from other people's sites is being withdrawn from you at the same time. You are being squeezed from both directions and neither squeeze will announce itself.
What Was Built on the Other Side
This is the half that does not get written about, and it is the half that matters.
In the same window that borrowed access was being fenced off, the permissioned rails got built and got adopted fast.
Anthropic open-sourced the Model Context Protocol in November 2024 and donated it to the Linux Foundation's Agentic AI Foundation on 9 December 2025. The foundation reports more than 10,000 published MCP servers, and its platinum members include AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, and OpenAI, with Salesforce, SAP, Snowflake, Shopify, Oracle, IBM, Okta, Datadog, Twilio, and Zapier at the gold tier. Bloomberg's CTO has described MCP as a foundational building block for APIs in the agentic era.
Payments went the same way. OpenAI launched Instant Checkout and the Agentic Commerce Protocol on 29 September 2025 with Stripe, starting with US Etsy sellers and over a million Shopify merchants to follow, with the merchant staying merchant of record. Google announced the Agent Payments Protocol in September 2025 with more than 60 partners including Adyen, American Express, Etsy, Intuit, Mastercard, PayPal, and Worldpay, then donated it to the FIDO Alliance in April 2026.
Even bot identity is being formalized. Web Bot Auth, now in the IETF, has agents cryptographically sign their HTTP requests using a published key, so a site verifies who is actually calling rather than trusting a user agent string that anyone can type.
And licensing became a market. Cloudflare's own count is more than 50 publisher and AI company agreements signed since 2023, including the New York Times with Amazon in May 2025, and OpenAI with Axel Springer, the Financial Times, Vox Media, The Atlantic, Condé Nast, Hearst, The Guardian, and Reuters. Really Simple Licensing launched in September 2025 letting publishers put machine-readable royalty terms directly into robots.txt, with Reddit, Yahoo, Quora, O'Reilly, and Medium as launch supporters.
Here is the thing. Every single one of those rails requires identity, permission, and usually a contract. Not one of them is scraping. The industry did not decide agents should have less access. It decided access should be accountable. And the businesses that will benefit are the ones with something to connect.
Practical Steps
1. Sort your data sources into owned and borrowed. Open every AI workflow you run and list what each one reads. Your CRM, your invoices, your email history, your past proposals, your booking system are owned. A competitor's site, a directory, a review page, a supplier catalog are borrowed. Anything that runs entirely on borrowed sources is on a clock.
2. Assume each borrowed source disappears tomorrow and price the loss. For each one, write down what stops, who notices, and how quickly. You are looking for the single loss that would hurt most. That is the one you fix first, and you fix it before it breaks, not after.
3. Check whether an official door exists. For several of your borrowed sources, there is a sanctioned API, a data feed, or a licensed product you have never looked at. It costs money and takes longer to set up. It also does not vanish on a Tuesday. I am not going to tell you how to get around a block, because the businesses doing that are the ones getting sued and delisted, and because it is the wrong foundation for anything you intend to keep.
4. Inventory the data you already own and never use. This is where most owners find something. Years of invoices. Every support ticket. Every sales call note. Every proposal you ever sent, including the ones that lost. Most people are chasing external data while sitting on a decade of their own that nobody has ever queried.
5. Connect one system properly instead of pasting exports. Pick the business question you currently answer by opening three systems and piecing it together. Connect that one system to your AI through MCP or a first-party API so it reads live data. Copy and paste is a workflow that only works when you are at your desk.
6. Write the context the numbers do not carry. A number without context produces an answer that is technically correct and practically useless. Before you connect anything, write down your seasonality, your typical deal size, your sales cycle, and the things that look like problems but are normal for your business. Otherwise you will get confident wrong conclusions, which are more expensive than no conclusions.
7. Set read and write as two separate decisions. Most people grant both at once because the connection flow asks once. Decide separately what the AI may do on its own and what it may only recommend. Start read-only. Here is a prompt for that specific decision:
[The Job]
Define exactly what my connected AI is allowed to do on its own and what it may only recommend to me.
This is for: me, setting the boundary before I turn the connection on.
It matters because: read access and write access are completely different risk decisions and I am about to grant both at once.
[The Background]
Here is what you need to know: the system I am connecting is [WHICH ONE] and the AI could potentially [WHAT ACTIONS IT COULD TAKE]. The records in it matter because [WHY]. If a record were changed incorrectly the consequence would be [WHAT].
Do not use: a permissive default. Assume read-only unless I justify otherwise.
[The Deliverable]
Return: a permissions table listing each possible action as allowed, requires approval, or blocked, with reasoning.
Must include: what a mistaken write would cost me and whether it is reversible.
Optimize for: reversibility.
[The Questions]
Ask me any questions you have.
Frequently Asked Questions
Is web scraping legal?
It depends far less on scraping law than people assume. In hiQ Labs v. LinkedIn, the Ninth Circuit found that scraping public data was unlikely to violate the Computer Fraud and Abuse Act, but the case ended in a $500,000 consent judgment against hiQ for breach of LinkedIn's user agreement, plus a permanent injunction. In August 2026, Amazon v. Perplexity narrowed the CFAA further. So federal computer crime law is a weak lever in both directions. Terms of service, copyright, and network-level blocking are the strong ones. You can win the CFAA argument and still lose on contract and access.
What is MCP and why should a business owner care?
Model Context Protocol is an open standard for connecting AI models to tools and data, open-sourced by Anthropic and now governed by the Linux Foundation. Practically, it is the difference between an AI guessing about your business from whatever it scraped and an AI querying your actual CRM, catalog, or bookings through a connection you control, with permissions you set and logs you keep. It is the sanctioned door. Scraping is the window.
How do I connect my own data to AI without a technical team?
Three layers, cheapest first. Put durable context in files the model reads directly: pricing, policies, positioning, past work. Then connect one system of record through an MCP server or a first-party integration so the AI reads live data instead of a stale copy. Then govern it, because an agent that can read your data can usually also act on it. Most small businesses get real value from the first layer alone and never do it.
My competitors are still scraping and it works for them. Why should I stop?
Because the trend line runs one direction and enforcement is landing at the infrastructure layer where individual sites have no say. Cloudflare delisted Perplexity as a verified bot in August 2025 after publishing evidence of undeclared crawling. Reddit sued Perplexity and three scraping intermediaries in October 2025. Major retailers have begun requiring AI agents to identify themselves in their terms. The question is not whether it works today. It is whether you want a core business process whose owner can switch it off and never tell you.
What is first-party data and why does it matter now?
It is information your business collects directly from its own customers and operations: transactions, support tickets, bookings, email lists, call notes, product usage, your own content. Be careful here, because the internet is full of unsourced statistics about first-party data value and I could not verify the popular ones. The defensible argument is structural, not statistical. Training and retrieval data is getting scarcer and more expensive, and a licensing market has formed with more than 50 publisher agreements signed since 2023. In that market, the business holding proprietary data holds an asset. The business renting access to someone else's holds a liability.
The Close
The thing I keep coming back to is not the block. It is the shape of the failure.
A blocked agent does not hand you an error message. It hands you a run that completed, with nothing useful in it. And somewhere between the morning it stopped and whenever you finally notice, you are making decisions on an empty result that looks exactly like a real one.
That is the actual risk, and it is not really about scraping. It is about the difference between a process that tells you when it stops and one that does not, and borrowed access can never be the first kind, because the person who turns it off has no obligation to tell you.
Here is the thing. You have been sitting on the answer the whole time. Every invoice you sent. Every support ticket. Every sales conversation where someone told you exactly why they did not buy. Years of it, in systems you already pay for, that nobody has ever asked a question of.
Nobody can revoke that. Nobody can reprice it. No terms of service update can fence it off from you. It is the one input in your business with no gate in front of it, and most owners are chasing someone else's data while ignoring a decade of their own.
The web got priced this year. Your own records did not.
Go use them.
About the author
Jonathan Mast is the founder and CEO of White Beard Strategies, where he teaches non-technical entrepreneurs how to use AI to amplify the skill and experience they already have. He runs a Facebook community of more than 500,000 members, the AI Insiders membership, and a live training program built on the Perfect Prompt Framework. He has spent enough years building businesses on other people's platforms to have strong feelings about owning the ground you stand on.