AI crawlers and bots

How many crawler hits you need before llms.txt can show a change

By Emily Parker · Nov 7, 2026 · 9 min read

What is published, and what is missing

Two kinds of evidence get quoted when people say llms.txt "works" or "does nothing." Neither is a before and after.

The first is a cross-section of many sites at one moment. Ahrefs analyzed all 137,210 domains in Ahrefs Web Analytics that received traffic in May 2026. About 28% of them, roughly 38,000 domains, published a valid llms.txt file. Of those, 97% saw no requests for it at all that month. The remaining 3%, about 1,100 domains, received all the measured traffic to the file. Ahrefs published the study on Jun 15, 2026.

That is a comparison of sites with other sites. It cannot tell you what happens to one site's crawler traffic when it adds the file, because the sites that got requests may differ from the sites that did not in many other ways. Ahrefs also says its customers skew technical, so the 28% adoption figure is an upper bound, and it did not check whether each file followed the spec.

The second kind is a statement from a search engine. Google's documentation on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." That page does not mention llms.txt by name. In a Reddit reply covered by Search Engine Journal on Jun 2, 2026, Google's John Mueller said that he doesn't think anyone knows whether the file matters, and called it "purely speculative for now." Both are statements about Google. Neither measures what other crawlers do.

In our search we did not find a study that took one site, added the file on a known date, and counted crawler requests on both sides. If you know of one, the method below is how you would check it against your own logs.

Who actually asked for the file

The Ahrefs study does break down who requested the files that were requested at all. This is the closest thing to a "what changed" answer, because it shows which visitors the file attracts. It is not the visitors you would want it to attract.

21.7%14.9%13.1%11.6%10.5%5.8%5.3%3.6%2.7%1.1%SEO audit toolsOther and unidentifiedGeneral web crawlersTech profiling toolsAI agents & agentic infrastructureGEO/AEO toolsAI training crawlersllms.txt discoverability botsResearch botsAI retrieval bots

Source: Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Jun 15, 2026. Retrieved Nov 4, 2026.

SEO audit tools made 21.7% of the requests. Unidentified bots made 14.9%, general web crawlers 13.1% and tech profiling tools 11.6%. AI agents made 10.5%, AI training crawlers 5.3%, and AI retrieval bots 1.1%. About 96% of requests to the files that received any traffic came from bots, and 4% from humans. GPTBot was the top named AI fetcher.

Two cautions go with those numbers. A fetch does not prove anything read or used the file, so these are ceilings on consumption. And Ahrefs could not determine the purpose or operator of the bots in the unidentified group, so that group could include AI traffic.

For a reader deciding whether to add the file, the useful reading is narrow. The first thing that shows up in your logs after you publish llms.txt is more likely an audit tool or a general crawler than an assistant that is answering someone's question. AI crawlers also do not probe for the file blindly: Ahrefs found no AI bot requests for llms.txt URLs that returned a 404. So publishing the file looks like a precondition for an AI crawler fetching it, which is the only part of this you control.

Why a before and after is hard to read on a small site

Suppose you add the file, wait four weeks, and compare your AI crawler hits to the four weeks before. What size of change would you be able to see?

Crawler hits arrive as discrete events, and the simplest model for counts like that is a Poisson process, where the spread of a count is about the square root of the count. Take two windows with n hits each. The difference between them has a spread of about the square root of 2n. A change has to be larger than about two of those spreads before you should believe it. Our arithmetic gives this:

Hits in each windowSmallest change you can tell from noise
25about 57%
100about 28%
400about 14%
1,600about 7%
10,000about 3%

This is arithmetic from a textbook model, not a measurement of any site. Real crawler traffic is burstier than Poisson. One crawler can fetch 300 pages in an hour and then vanish for a week, which makes the true noise larger and the table optimistic.

A single small paper document set at the edge of a quiet country road at dawn, with only a few small robot-like figures walking past in the distance, one pausing to glance at it.

So the practical question is not "did the total move." It is "do I have enough hits for a total to mean anything." At about 100 AI crawler hits in four weeks, a doubling would be visible and a 20% rise would not. The file's own request count is a better test at that size, because for AI crawlers it starts from zero: in the Ahrefs data, no AI bot asked for a file that was not there.

What else moves your crawler count that month

If you do have enough volume, the comparison still has to survive other changes. Keep a list of everything you touch in the baseline and test windows, because each of these can swamp the effect of a text file:

  • robots.txt edits. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. A rule change in the same week as llms.txt will move counts for reasons unrelated to the file.
  • New pages. A batch of new URLs or a new sitemap draws crawlers whether or not llms.txt exists. Our post on which AI crawlers reach a new site in its first 30 days shows how much a young site's crawler picture changes on its own.
  • A launch or a link. A mention somewhere popular sends scanners and crawlers to you for a few days.
  • Scanners posing as crawlers. One tool can make a count look like a trend. The log check for a scanner in costume tells you whether a spike is one client.
  • A growing crawler list. If the list of recognized agents grows between your two windows, new hits get a label that old hits never had. That is a bookkeeping change, not a traffic change.

The last item is where the tool you count with matters. Peeko matches each crawler user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. It keeps the raw user agent, up to 512 characters, so old hits can be reclassified as the list grows. That lets you compare a baseline and a test window against the same list instead of against two different versions of it.

Run it in three steps

You do not need the whole experiment to learn the most important number. Start with the file itself.

Step 1: count requests for the file in your access log. On nginx or Apache with the combined log format, this lists who asked for llms.txt or llms-full.txt, by user agent:

grep -E '"GET /llms(-full)?\.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

If you have been serving the file for a month and this prints nothing, that matches what Ahrefs saw on 97% of sites. A CDN may answer some requests before they reach your origin, and rotated logs need checking, so count from every log you have.

Step 2: see when the requests came, by day.

grep -E '"GET /llms\.txt' access.log | awk '{print substr($4,2,11)}' | uniq -c

One burst on a single day from one user agent is a scanner. Requests spread across weeks from a named AI crawler are the thing worth acting on.

Step 3: compare crawler hits by category, but only if the volume supports it. Write down your four-week total for each category before the change date. Check the table above against that total. If your count in a window is under about 100, treat the category totals as a record, not a test. Our fuller before and after method lays out the same experiment with a longer baseline.

What Peeko can and cannot tell you here

AI crawler tracking is a separate server side path, because crawlers do not run JavaScript. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. No address and no visitor hash exist on that path at all.

Three properties matter for this experiment:

  • Crawler hits are never billed. An event is a page view, an outbound click or a custom event, so a month of extra crawling does not push you toward your limit.
  • Records are kept three years by default. A site owner can set the window from 30 days to ten years, so the baseline you need will still be there when you decide to test.
  • The command line reads the same numbers as an agent. npx @usepeeko/cli speaks the same MCP endpoint, so you can ask your coding agent for the crawler totals instead of exporting them.

What we are not claiming: Peeko files hits into four categories, and for the file itself you should use your access log as in step 1. Do not read a category total as a count of llms.txt fetches.

What we did not measure

We did not run a before and after on our own site for this post, so there is no Peeko result to report. Every request count above comes from the Ahrefs study, and it covers one month. The table is arithmetic from a simple model. We also have no data on whether an assistant used a file it fetched, and the question a person asks an assistant never reaches your server, so nobody can report that from the site side. For the effect on visits, the part you can count is visits arriving from assistants by referrer, covered in how to count visits from ChatGPT, Claude, Perplexity, Copilot and Gemini.

A decision rule

Writing the file takes under an hour for a small site, and the guide to writing llms.txt for a small product site covers it. The reading side is cheaper if you set it up first:

  1. Publish the file and note the date.
  2. Count requests for it in your log after one week and after four. Zero is a result, and it is the most likely one.
  3. Compare crawler totals only if each four week window holds at least 400 hits, which makes a change of about 14% visible. Below 100, do not draw a conclusion from the total.
  4. Keep the file if a named AI crawler fetches it more than once, or if developers and agents that read your docs use it. Drop it from your to do list if nothing has asked for it after eight weeks and you have other work.

Do not expect the file to raise your crawler hits. The published evidence shows nobody has demonstrated that, and on the sites that get requests, SEO audit tools are the largest single source. The cost of finding out on your own site is one file and a log search.