Which crawlers actually request llms.txt, and how often

You added a file at /llms.txt and now you want to know whether anything ever asks for it. The honest answer is: rarely, and mostly not the companies you were hoping for. This post goes through the one large measurement that exists, what the AI companies say their crawlers do, how to check your own server logs in two commands, and what we can and cannot tell you about it.

We did not run an llms.txt experiment of our own for this post. Everything below about request counts comes from the study we cite, and we say where its limits are.

What the largest log study found

Ahrefs looked at every domain in Ahrefs Web Analytics that received traffic in May 2026: 137,210 domains. About 28% of them (38,360 domains) published a valid llms.txt file. Of those files, Ahrefs reports that "97% of those files received zero traffic in May. Nothing fetched them at all."

That leaves roughly 3% of files that anything requested. Here is who made those requests, as a share of all the llms.txt requests Ahrefs recorded.

Source: Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 2026-06-01. Retrieved Oct 8, 2026.

Three things stand out.

First, the top of the list is not AI. SEO audit tools made 21.7% of requests, other and unidentified bots 14.9%, general web crawlers 13.1%, and tech profiling tools 11.6%. Those four categories alone are about 61% of requests. Ahrefs puts 96% of the traffic to these files in the bot column, and 4% with humans, so a person opening the URL in a browser is rare too.

Second, the AI categories are small. AI training crawlers account for 5.3% and AI retrieval bots for 1.1%. GPTBot, OpenAI's training crawler, is the largest named AI fetcher at 4.51%. Ahrefs adds that OAI-SearchBot, PerplexityBot and Claude's search crawler "combined made only a couple of hundred fetches" across the whole dataset.

Third, AI bots do not go looking for the file. Ahrefs also checked requests for /llms.txt that came back 404, and AI bots made none of them.

Limits of that measurement

Read the number with its edges in view.

• The sample is domains that run Ahrefs Web Analytics. It is large, but it is not a random sample of the web, and small personal sites may behave differently.
• It covers one month. A crawler that changes behavior in June would not show up.
• Bots were classified by user agent and checked by hand into twelve categories. A user agent is a claim the client makes about itself, so some "unidentified" traffic is probably AI traffic under another name.
• Ahrefs says it plainly: "A bot requesting your llms.txt isn't proof anything read or acted on it." Fetched is the upper bound on read.

What the AI companies say their crawlers are for

The documentation of the major AI companies describes crawlers by job. None of the pages we read describes llms.txt as an input.

OpenAI lists three user agents. OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features." GPTBot is "used to crawl content that may be used in training our generative AI foundation models." ChatGPT-User is "used for certain user actions in ChatGPT and Custom GPTs," and is not used for crawling the web automatically.

Anthropic also splits its crawlers by job. ClaudeBot collects web content that could contribute to training. Claude-User supports people who ask Claude a question and may visit a site for the answer. Claude-SearchBot improves search result quality. All three honor robots.txt directives. The page does not mention llms.txt.

Both companies publish an llms.txt for their own documentation. OpenAI's docs open with "For the complete documentation index, see llms.txt," and Anthropic's docs file answers at platform.claude.com/docs/llms.txt. That tells you the companies think the format is useful for sites that serve developers. It does not tell you their crawlers request yours.

Google is more direct. Its page on AI features in Search says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." It adds that there are no additional requirements to appear in AI Overviews or AI Mode.

Why a few AI requests still show up

The pattern in the Ahrefs data fits what the documentation describes. Training crawlers sweep URLs they have found. Search crawlers build an index. User-triggered fetchers, like ChatGPT-User and Claude-User, visit a page because somebody asked about it. Each of these has a reason to request a page you linked. None of them has a documented reason to request a special file at the root.

The one place AI shows up in the Ahrefs list as a real presence is the "AI agents and infrastructure" group at 10.5%, and Ahrefs names Claude-Code as exceeding every AI retrieval bot and assistant. If you built your site with an AI coding agent, that is the likely reader of your llms.txt: an agent working for a developer, not a crawler building an index. That reader is real, but it is one person's session at a time, and it arrives rarely.

Check your own logs

You do not need an analytics product to answer your question. If your server writes the common combined log format (nginx and Apache both do by default), this prints every client that requested the file, with a count:

grep -E '"GET /llms(-full)?\.txt' /var/log/nginx/access.log \
  | awk -F'"' '{print $6}' \
  | sort | uniq -c | sort -rn

The sixth quote-delimited field is the user agent. To see how the file compares to robots.txt on the same site, count both:

for f in robots.txt llms.txt; do
  printf "%s: " "$f"
  grep -c "\"GET /$f" /var/log/nginx/access.log
done

Three cautions before you trust the numbers:

1. If a CDN answers from cache, the request may never reach your origin log. Check the CDN's own logs as well.
2. Rotated logs hold older days. Run the same commands on access.log.1 and the compressed archives with zgrep.
3. Anything that matches GPTBot or ClaudeBot in the user agent is a claim, not proof. Operators publish IP ranges you can check against; the post on naming the operator of an AI agent without IP addresses covers what you can and cannot establish when you do not keep addresses.

On a small site, expect the result to be empty or a handful of lines from audit tools. That matches what the large study found, and it is not a sign that your file is broken.

What Peeko can tell you here, and what it cannot

Crawlers do not run JavaScript, so a script tag never sees them. AI crawler tracking in Peeko is a separate server side path: the @usepeeko/bot-track middleware, which is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped.

It matches the user agent against a curated list of 55 agents from 21 companies and files every recognized hit under one of four categories: AI answers, Search indexing, AI training or Other AI bots. That is the same split the documentation above draws, and it is more useful for your question than a single "bot" total. A ChatGPT-User fetch and a GPTBot fetch mean different things, and the categories keep them apart.

A few details matter for a log question like yours:

• No address and no visitor hash exist on the crawler path at all, so there is nothing to leak or to subpoena.
• The raw user agent is kept up to 512 characters. When the crawler list grows, old hits can be reclassified, so an "unidentified" hit from last quarter can become a named agent later.
• AI crawler hits are never billed. An event is a page view, an outbound click or a custom event, so watching crawlers costs you nothing against your monthly limit.

We are not claiming that Peeko reports a per-file count for /llms.txt. For that exact question, the grep above is the right tool. What Peeko adds is the wider picture: which AI companies visit your pages at all, by category, over time. If your site is new, what to expect from AI crawlers in the first 30 days sets a realistic baseline before you read anything into an empty log.

One adjacent measure is worth separating out. A person who clicks through from an assistant is not a crawler hit at all. Peeko counts those visits from ordinary page views by their referrer, from ChatGPT, Claude, Perplexity, Copilot and Gemini, with no middleware involved. The post on counting visits from AI assistants by referrer explains it. Compare that number to your llms.txt count and you will see where the attention actually comes from.

So should you keep the file

The cost is one small text file, and the evidence says the audience is small. Keep it if it helps the agents and developers who read your site, and write it for them: a short description, links to the pages that matter, plain Markdown. Do not expect it to move you into AI answers, and do not let it displace the work that does show up in logs, like clear pages, working robots.txt rules and links from other sites.

What to take away, as of Oct 8, 2026:

• 97% of llms.txt files in a 137,210 domain study were never requested in May 2026.
• The requests that did arrive came mostly from SEO audit tools, unidentified bots and general crawlers, not from AI answer products.
• The AI companies document their crawlers by job and say nothing about reading llms.txt. Google says you do not need AI text files at all.
• Your own access log answers the question for your site in about ten seconds.