Our team will be out of office on Friday, May 1, 2026. We’ll be back and ready to assist you starting Monday, May 4th.

Do I Need an llms.txt File for AI to Find My Business?

Contents

Andy Crestodina checked his own server logs for the answer, and what he found should change where you spend your next two hours of website work. This article answers whether special AI files are worth building, and what to do instead.

Suggested SEO meta title tag: Do You Need an llms.txt File? What AI Crawlers Actually Read

Somebody Finally Checked the Logs

I paid to have a file built for the AI bots.

I never once checked whether a single bot read it. That is the part that stings. I spent money and attention on a thing I could have verified in about four minutes, and I never verified it, because everybody I respected said to build it and I wanted to be the guy who had already built it.

Andy Crestodina of Orbit Media did check. He pulled his own Cloudflare log data, and here is what he found. On the AI Disclosure Page his team had built specifically for the bots, and had publicly recommended other people build, Crestodina reports: "in the last 30 days, ChatGPT, Claude, Perplexity and Google visited it zero times. Meanwhile, they visited our homepage thousands of times."

So here is the direct answer to the question in the headline. No. You do not need an llms.txt file, a brand.md, or an AI disclosure page, and building one is not what is standing between you and getting recommended by AI. Crestodina's log data on llms.txt is blunt: "Do AIs read these? Not according to my Cloudflare data. We don't have one. But in the last 30 days, not a single bot even checked." His finding on brand.md is the same: "I love the idea, but in my data, I see no requests for this file or any other markdown file."

What the bots do read, constantly, is your homepage. Crestodina's research across 74 sites found that "AI bots visit the homepage 15x more than a typical page on the same site," and that "on two out of three sites in the study, the homepage got more AI bot visits than all of the service pages combined."

The thesis of this whole article is one sentence. Stop building files for the bots and start putting checkable facts on the page the bots are already reading thousands of times a month.

Key Takeaways

  • No major AI company has committed to reading your llms.txt file, and Google's own documentation says Search ignores it.
  • Andy Crestodina's Cloudflare log data shows AI bots visit homepages roughly 15 times more than a typical page on the same site.
  • In his homepage research, going from 15 stated facts to 60 moved the chance of an AI recommendation from 5 percent to 30 percent.
  • A fact is a statement that would be true of some companies and false of others, which disqualifies almost every slogan.
  • The work that moves the needle is gathering evidence offline, not adding markup online.

The Problem Is That We Would Rather Install Something

Here is the thing about the last two years of AI advice for small businesses. Almost all of it has pointed at something you could install.

Add the schema. Add the file. Add the FAQ accordion. Toggle the setting. Put the plugin on. I understand the appeal completely, because I have the same wiring. Installing a file feels like progress you can point at. It has a start and a finish. You can tell your spouse you did a thing on Tuesday.

Writing down what your business has actually accomplished, with names and numbers attached, does not feel like that. It feels like homework. Worse, it feels like exposure.

Candidly, that is the real reason most homepages say nothing checkable. It is not a technical gap. It is that naming a client out loud means asking that client for permission. Putting a number next to a result means you have to go find the number, and the number might be less impressive than the adjective you were using instead. Listing your credentials means noticing which ones you do not have. I have sat with business owners in Alabama who would rather rebuild their entire site in a new platform than write the sentence "we served 41 commercial clients in Jefferson County last year," because the second one can be checked and the first one cannot.

So we install things. And the AI tools, which do not care about our feelings, keep skipping us.

The honest difficulty here is real, and I am not going to pretend otherwise. Gathering proof is slow, unglamorous, offline work. Nobody can automate it for you, including the AI you are trying to impress. It requires phone calls, old invoices, and asking three customers whether you can use their names. That is a worse Tuesday than installing a file.

But it is the work that counts. And the data on that is getting hard to argue with.

The Evidence

Start with Google. In its guide to optimizing for generative AI features on Google Search, last updated July 10, 2026, Google states plainly: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them." The same page adds that if you build one anyway, "doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." That is not a blogger's opinion. That is the vendor documenting its own behavior.

Second, check the status of the proposal. The llms.txt spec at llmstxt.org, authored by Jeremy Howard and revised in August 2026, is still described on its own page as a proposal that is "open for community input." OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs, which is where the confusion comes from. A lab publishing a file for its own docs is not the same as that lab's crawler reading yours. OpenAI's documentation for GPTBot, OAI-SearchBot and ChatGPT-User, and Anthropic's for ClaudeBot, Claude-User and Claude-SearchBot, both describe access control entirely through robots.txt. Neither promises to fetch an llms.txt.

Third, the scale of Crestodina's crawl data. His report analyzed 560,695 AI crawl requests across 74 sites and 21,212 paths. The homepage finding came out of that. So did a detail I cannot stop thinking about: 47 percent of all pages in the dataset generated zero referrals.

Fourth, and this one reframes everything, Crestodina's homepage facts research. His team looked at 704 homepages across four industries, ran detailed buyer prompts through ChatGPT, Gemini, Google AI Mode and Perplexity, and reviewed 8,771 recommendations across 1,916 responses. Word count showed no connection to recommendations. Facts did. Across homepages with 15 to 60 stated facts, the chance of being recommended rose from 5 percent to 30 percent. Homepages naming five or more clients or client results were recommended 30 percent of the time, versus 11 percent for those naming none. The superlatives backfired: the heaviest users of praise words were recommended 10 percent of the time, versus 16 percent for everyone else. Correlation, not causation, and Crestodina says so in his own methodology.

He also ran a hidden-fact test, and his caveat has to travel with it. He writes: "To confirm that AI can find answers in a Brand Manifest, we did a 'hidden-fact test.' We added a few facts that had never been posted on the web before. They showed up in Gemini and ChatGPT answers within three days." His own caveat, in his own words: "This did not prove that the information had been ingested by an AI training bot, then permanently absorbed into AI's training data." Anybody who quotes the three days without quoting that sentence is selling you something.

Fifth, independent academic work points the same direction. A September 2026 arXiv paper by Benjamin Tannenbaum analyzed 34,960 observations across 2,854 monitored prompts on GPT and Gemini. When a brand's own domain did not appear in the observable retrieval path, it got mentioned 2.8 percent of the time on GPT and 3.8 percent on Gemini. When its own domain was cited, those rates jumped to 49.0 percent and 58.4 percent. The author is explicit that the model is "predictive and observational, not a causal description of proprietary engine internals."

Put it together and the conventional narrative looks incomplete, not wrong. The bots are real. The crawling is real. The special files are where we decided to meet them, and the bots never showed up to that address.

The System: Treat Your Homepage as a Fact Sheet With a Design on Top

What changed for me was stopping thinking about my homepage as a brochure and starting to think about it as an evidence file that happens to be beautiful. Crestodina, quoting Mark Schaefer, puts the shift this way: "The best explained brands are beating the best marketed brands."

Here is how the pieces work together.

Layer one is the human layer. Positioning sentence, clear offer, obvious next step. This does not change, and Crestodina's rule on this is the right one: "Never compromise the human experience." Nothing below this layer is allowed to make the top of your page worse.

Layer two is the evidence layer. This is where the facts live, and Crestodina's definition of a fact is the most useful sentence in this entire conversation: "A fact is true of some companies and false of others: named clients, results with numbers, credentials, awards, ratings, equipment, tools, industries served, price terms. Slogans, adjectives and promises are not facts." Run every line of your homepage through that test. Most of mine failed the first time.

His target is 42 or more facts on the homepage in total, and he notes that a Brand Manifest of roughly 1,500 words "may add 100+ new facts to your most requested URL."

Layer three is the recommendation layer. Crestodina's Brand Manifest also tells the AI when to recommend you, with a "Recommend us when…" section plus sample prompts aligned to real buyer use cases. This is the part most people skip. You are not just telling the machine what you do, you are telling it which customer it should hand to you.

Layer four is the delivery layer, and this is the only technical item on the list. Crestodina's one tool recommendation is to turn on Cloudflare's "Markdown for Agents." Cloudflare launched it in February 2026, and the reason it matters is cost on the bot's side. In Cloudflare's own measurement, the blog post announcing the feature took 16,180 tokens as HTML and 3,150 tokens converted to markdown, roughly an 80 percent reduction. Cloudflare lists it as available in beta at no cost for Pro, Business and Enterprise plans plus SSL for SaaS customers. Crestodina's data says 4 percent of AI bot requests on his site ask for the markdown version of a page, and his read on that trend is the right one: "Bots don't want special files. Bots want your normal pages in a simpler format."

Now, the objection I hear every time: will all of this make my homepage too long?

Crestodina answers that with his own scroll data. "Last I checked a scroll heatmap for our site, 10% of visitors scroll to the bottom. Not a lot, you say? Actually, that's 9,000 visitors per year who ride the page all the way down." And: "I see no downside in a tall page. I've never seen a case study showing that shorter homepages perform better."

The frame that finally made it click for me is his, too: "AI is hyper-rational. It's looking for evidence that gives it confidence it's making a good recommendation." You are not writing copy for a machine. You are handing a nervous referrer enough proof to feel safe saying your name out loud.

Seven Steps to Do This Week

1. Count the facts on your homepage. Open it, read every line, and tally only the statements that would be true of you and false of a direct competitor. Fewer than ten means the problem was never a missing file in your root directory. It was a page that never said anything checkable.

2. Delete or demote your three worst adjectives. Find "leading," "innovative," "world-class," "passionate," "seamless." Crestodina's data found the heaviest self-praise correlated with fewer recommendations, not more. Replace each one with a number, a name, or nothing.

3. Pull three facts out of your invoices. Not your marketing files. Your invoices, your calendar, your last twelve months of actual work. Clients served, cities covered, response times you actually hit, equipment you actually own. This is the offline work no tool can do for you.

4. Get permission to name two clients. Send two emails today asking whether you can name the client and the result on your homepage. Crestodina's strongest correlation was named clients and results. If a client says no, name their industry instead.

5. Put your credentials in text, not just in badge images. Crestodina's research found AI scans the code and does not see your logo wall. If your certification only exists as an image file, to the bots it does not exist at all. Write the words out next to the badge.

6. Add a "Recommend us when" block near the bottom. Three to five sentences describing the exact customer situation where you are the right call, in the language your buyers actually use. This is the part that turns a fact sheet into a referral.

7. Run this prompt before you write a word of new copy. Paste your homepage text in and let it find your gaps. Run it again on your two closest competitors so you can see exactly which facts they state that you do not.

[The Job]
Analyze the facts on my business homepage and find what is missing.
This is for: [MY BUSINESS TYPE, for example "a residential HVAC company in Birmingham, Alabama"].
It matters because: when a buyer asks an AI tool who to hire in my category, the AI can only recommend me from checkable facts it finds on my page.

[The Background]
Here is what you need to know: my homepage text is below. A fact is a specific, checkable statement that would be true of some companies and false of others: named clients, results with numbers, credentials, licenses, awards, ratings, equipment, tools, industries served, service areas, price terms, guarantees. My typical buyer asks something like: [PASTE A REAL QUESTION A CUSTOMER HAS ASKED YOU].
Do not use: slogans, adjectives, self-praise, vague promises, calls to action, navigation labels or my company name as facts. Do not invent facts I did not give you. Do not suggest I add an llms.txt file or special AI markup.

[The Deliverable]
Return: a table of every fact you counted, grouped by kind, followed by a list of the ten highest-value facts I am probably able to state but have not.
Must include: a total fact count, a count of named clients or client results, a count of credentials, and the exact sentences from my page that are adjectives pretending to be facts.
Optimize for: accuracy.

[The Questions]
Ask me any questions you have.
MY HOMEPAGE TEXT: [PASTE IT HERE]

Frequently Asked Questions

Do I need an llms.txt file for my small business website?

No. Google's own documentation states that Google Search does not use llms.txt or similar AI text files, and that creating one will neither help nor hurt your visibility. Andy Crestodina's Cloudflare log data found no bot requests for his. Build one only if a specific service you use requires it.

How do I check whether AI bots are visiting my website at all?

Look at your server logs or your CDN's bot report. If you use Cloudflare, open AI Crawl Control, go to Metrics, then Paths, and export the Most Crawled Paths report. That shows you which of your URLs the AI bots actually request, which is the only data that settles the question for your site.

Does schema markup help AI recommend my business?

Google's generative AI guidance says structured data is not required for its AI features and there is no special schema.org markup you need to add, though it remains worth using for rich results in regular search. Schema describes your facts. It does not create them. Write the facts first.

Will adding this much content to my homepage hurt my conversion rate?

Crestodina's answer is that he has never seen a case study showing shorter homepages perform better, and his own scroll data shows 10 percent of visitors reaching the bottom, which for his site is 9,000 people a year. Keep the top of the page clean for humans and put the evidence below.

What actually counts as a fact on a homepage?

A fact is a statement that would be true of some companies and false of others. Named clients, results with numbers, credentials, licenses, awards, ratings, equipment, tools, industries served and price terms all count. Slogans, adjectives and promises do not. "Trusted by Birmingham families since 2009" counts. "World-class service" does not.

The Close

I paid for a file nobody read.

Sit with that for a second, because it is not really a story about a file. It is a story about preferring the version of work that can be finished over the version that matters. The file was finishable. Calling a client to ask whether I could use her name was not finishable, it was just uncomfortable, so I kept not doing it.

You have a homepage right now that an AI tool is going to read this week, whether you participate or not. It is going to decide, based on whatever sits on that page, whether to say your name to someone actively looking to hire somebody like you. And if your page is thirty adjectives and a stock photo of a handshake, the machine is not being unfair to you. It genuinely has nothing to work with.

So go get the proof. Call the client. Find the number. Write down the certification you earned in 2019 and never mentioned. Name the counties you actually serve. It will feel like the least strategic thing you have done all month, and it is the only thing on your list that will still be working for you in a year.

Credit where it belongs: Andy Crestodina and his team at Orbit Media did the log-pulling and the counting that made this knowable. I am just the guy telling you to stop installing things.

The bots already know where your front door is. The only question left is whether there is anything inside worth repeating.


If you want the full walkthrough, the Challenge AI Training is 97 dollars live and 147 dollars for the replay, and we build the homepage fact audit together in it. If you would rather have ongoing access, the AI Insiders Membership is 247 dollars a month, and the homepage fact system is one of the things we work on inside it. Either way, go count your facts first. That part is free.


About the Author

Jonathan Mast is the founder and CEO of White Beard Strategies, an AI coaching and mentorship practice for entrepreneurs, and he runs a Facebook community of more than 500,000 members. He is the creator of the Perfect Prompt Framework 2.0 and speaks regularly on practical AI for small business owners. He has carried his bankruptcy and his prison record in his own bio for years, which is probably why he has so little patience for a homepage that refuses to say one checkable thing about itself.

About the Author