What the data says happens to crawler traffic after llms.txt

You want to know what happens to crawler requests after you add an llms.txt file, and you want the answer before you spend time on it. The honest answer has two parts. The best public measurement says the file itself is almost never fetched. And no public measurement we found shows that adding it changes how often AI crawlers visit the rest of your pages.

This post covers the one large dataset we could verify, what it can and cannot support, and a method you can run on your own site in a month.

What the largest public measurement found

Ahrefs looked at 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026. It checked each domain root for an llms.txt that returned HTTP 200, and it excluded soft 404s by confirming the file held real Markdown rather than an error page. More than one in four domains, about 38,000, had one.

Then it counted who asked for those files. Its summary: "97% of those files received zero traffic in May 2026. Nothing fetched them at all." Only about 3% of the files, roughly 1,100 domains, saw any request.

The requests that did arrive came mostly from tools that have no stake in AI answers. This chart shows the share of all llms.txt requests by bot category.

Source: Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 2026-05. Retrieved Oct 8, 2026.

SEO audit tools were the largest group at 21.7%, and the category the study labeled "other and unidentified" was next at 14.9%. AI retrieval bots, the ones that fetch pages to answer a live question in an AI search product, were 1.1%. AI training crawlers were 5.3%. AI assistants, which browse on behalf of a user, were 2.5%.

Read the chart carefully. These are shares of requests to the file among the 3% of sites that got any. They are not shares of all AI crawler traffic, and they say nothing about how often the same bots fetched ordinary pages.

What the 97% does and does not mean

The number answers a narrow question: does anything request /llms.txt? For 97% of sites that published one, nothing did in that month.

It does not answer the question you likely care about, which is whether adding the file makes AI crawlers visit more, visit different pages, or cite you more. To answer that you would need the same site measured before and after, with nothing else changed. The Ahrefs study is a one month snapshot across many sites. It compares sites with sites, not a site with itself.

Three limits are worth stating in the same breath as the numbers:

• It covers May 2026 only. A crawler that adopts the file in June would not show up.
• Ahrefs says its customers "skew more technical and SEO-aware than the web at large," and calls the roughly 28% adoption figure an upper bound. Your site may sit in a different population.
• The sites are those running Ahrefs Web Analytics, so the result reflects what that tool can see.

Ahrefs draws its own conclusion from the data: "If your goal is showing up in ChatGPT, Perplexity, or AI Overviews, an llms.txt file is largely decoration." That is their reading, and it fits the numbers. It is still a claim about the file being requested, not a controlled test of ranking or citation.

Why the retrieval bots matter most

Not every AI bot does the same job, and the difference decides what you would hope the file does. OpenAI's own crawler documentation splits its agents by purpose:

• OAI-SearchBot "is used to surface websites in search results in ChatGPT's search features."
• GPTBot "is used to make our generative AI foundation models more useful and safe."
• ChatGPT-User: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent."

The documentation is on OpenAI's bots page. If an llms.txt were going to help you appear in answers, the agents that read pages while somebody waits for a reply are the ones that would need to fetch it. In the Ahrefs data, the category built for that job was the smallest of the twelve, at 1.1% of file requests. It said that OAI-SearchBot, PerplexityBot and Claude's search crawler "combined made only a couple of hundred fetches across thousands of sites."

That does not mean those bots will never read the file. It means that in May 2026 they mostly did not.

How to measure your own before and after

Crawler traffic is noisy. A single new blog post, a robots.txt edit or a link from a popular page can move it more than a text file can. So a before and after only means something if you control what you change.

1. Pick a start date and change nothing else. Publish /llms.txt on one day and write the date down. Do not edit robots.txt, restructure URLs or ship a large batch of new pages in the same window.
2. Record a baseline first. Count four full weeks of crawler hits before the date. Our post on what AI crawlers do on a new site in the first 30 days shows what a normal baseline looks like and why the first weeks are lumpy.
3. Count requests for the file itself. If you have raw access logs, this works on a standard nginx or Apache combined log:

grep -c '"GET /llms.txt' /var/log/nginx/access.log

4. Break those requests down by user agent. The user agent is the sixth quote-delimited field in the combined log format:

grep '"GET /llms.txt' /var/log/nginx/access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

5. Count crawler hits on every other page, by week. Compare the four weeks after your date with the four weeks before, by crawler type, not in total. A rise in training crawlers and a flat line for retrieval bots are different stories.
6. Wait for a meaningful count. If the file gets 0 requests in four weeks, that is a result. It matches what Ahrefs saw for 97% of sites. If it gets 3, do not read a trend into it.

Keep the claim small. One site and eight weeks can tell you whether anything fetched the file and whether your overall crawl pattern moved. It cannot prove the file caused the movement.

Where raw logs fall short

Logs work if you run your own server. Many small sites and products do not get raw logs, and crawlers do not run JavaScript, so a script tag never sees them. That is why we built AI crawler tracking as a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped:

npm install @usepeeko/bot-track

Peeko matches each request's user agent against a curated list of 55 agents from 21 companies. It files every recognized hit under one of four groups: AI answers, Search indexing, AI training or Other AI bots. Those groups map onto the question above. You can watch AI answers separately from AI training, and see whether either moved on your start date.

Two details matter for a before and after test:

• The raw user agent is kept up to 512 characters, so old hits can be reclassified as the crawler list grows. A bot you could not name in September can be named in December, and the September hits move with it.
• Crawler hits carry no IP address and no visitor hash at all. The path exists to count agents, not people. For how that works when you want to name the company behind a request without addresses, see naming the operator of an AI agent without IP addresses.

AI crawler hits are never billed. An event for billing is a page view, an outbound click or a custom event, so a month of heavy crawling does not use up your allowance. Page views, events and crawler hits are kept three years by default, which is long enough to compare a quarter before the file with a quarter after.

There is also a second signal to watch. A crawler fetching a page is one thing. A person arriving from an assistant is another, and Peeko counts those from ordinary page views by their referrer, with no middleware. If you want to see whether answers are sending people your way after the date, read how to count visits from AI assistants by referrer. An answer fetch says a page was read while an assistant was replying to somebody. The question itself never reaches your server, so nobody can report it.

What we did not measure

We are not publishing a Peeko before and after result for llms.txt in this post. The numbers above come from Ahrefs, and we have cited them as theirs. What Peeko adds is the method and the instrument: a way to separate answer fetches from training crawls on your own site, without keeping an address.

A decision rule

Add the file if it takes you a short, bounded amount of time, and treat it as an experiment, not a fix. Write the date down. Count the requests for the file and the weekly hits by crawler type for four weeks on either side.

If nothing fetches it, you have learned what 97% of the sites in the Ahrefs data learned, and you spent little. If a crawler you care about starts fetching it, you have a reason to maintain it. Either way, the number you trust will be the one from your own logs, not one from a study of other people's sites.