AI crawlers and bots
Writing an llms.txt for a small product site
By Theo Marsh · Oct 26, 2026 · 9 min read
What the file is, and what the evidence says
An llms.txt file is a Markdown index of your site at one fixed address. Ahrefs, which studied the format, calls it "a single index file, written in markdown, placed at a site's root" that "summarizes what a site is and links its most important content." It is not a robots.txt style directive. In Ahrefs' words, "it controls nothing and blocks nothing."
Start with the search engine that matters most to a small site. Google's AI optimization guide, last updated Jul 10, 2026, says "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search" and that "Google Search itself doesn't use them." It also says "It's completely fine if you decide to create and maintain LLMS.txt files" for other services, and that doing so won't affect your visibility in Google Search.
Then look at who asks for the file. Ahrefs studied 137,210 domains that run Ahrefs Web Analytics and had traffic in May 2026. Of those, 28% published an llms.txt, and 97% of those files received zero traffic that month. Across the files that were requested at all, 96% of requests came from bots. The chart shows which kinds of bots.
Source: Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Jun 15, 2026. Retrieved Oct 23, 2026.
Read the chart with its limits in view. Ahrefs says its customers skew more technical than the web at large, so 28% adoption is an upper bound. The data covers one month. A request is not proof that anything read or used the file. Ahrefs also did not check whether the files it counted were well formed.
What it adds up to: SEO audit tools fetch the file far more often than AI retrieval bots do, and AI training crawlers fetch it nearly five times as often as retrieval bots. We have no before and after result of our own for llms.txt, and this post does not claim one. The file costs about 30 minutes. Write it as a map for an agent that is already on your site, not as a ranking lever, and do not iterate on it for weeks. For the full breakdown of who fetched what, see which crawlers request llms.txt and how often.
What goes in it
Use this shape, top to bottom:
- A name. One H1 with your product name.
- A summary. A blockquote of two sentences: what the product is, who it is for, and one fact a reader could check, such as the price or a limit.
- A short paragraph of context. Optional. Use it for anything a link note cannot carry, such as which mail providers you support.
- Sections of links. H2 headings, each with a list of
- [Title](URL): notelines. Peeko's own file and Perplexity's documentation index both use that line format. - A last section for skippable links. The llms.txt format names it Optional, for links a reader can skip. Put the blog and anything else a reader can do without there.
Pick the pages a stranger would need to decide whether your product fits and then use it: what it does, pricing, getting started, rules or syntax, limits, a changelog, and your privacy page. Aim for 10 to 20 links. Our file is longer because it also lists our posts and tools, and yours probably does not need that. A file that lists every page is a sitemap, and you already have one.
Write each note as the question the page answers, with a number where one exists. "Rule counts, mailbox counts, and sync delay" tells a reader more than "Learn about our limits."
If you can serve a Markdown copy of your key pages, link those. Peeko and Perplexity both do, with a .md ending on the URL. If you cannot, link the normal HTML pages. Here is a complete file for an invented product. Every fact in it is made up.
# Tidyfox
> Tidyfox sorts a freelancer's inbox with rules written once in plain language. It works with Gmail and Outlook in the browser. One plan, 8 dollars a month, 14 day free trial with no card.
Free accounts are limited to 3 rules. Tidyfox does not send mail on your behalf.
## Product
- [What Tidyfox does](/about.md): The sorting model, the two supported mail providers, and what it cannot do.
- [Pricing](/pricing.md): The one plan, the trial terms, and what happens when a trial ends.
- [Changelog](/changelog.md): Dated list of changes, newest first.
## Docs
- [Getting started](/docs/start.md): Connect a mailbox and write a first rule in about 5 minutes.
- [Rule syntax](/docs/rules.md): Every condition and action a rule accepts, with examples.
- [Limits](/docs/limits.md): Rule counts, mailbox counts, and sync delay.
## Policies
- [Privacy](/privacy.md): What is stored, for how long, and how to delete it.
## Optional
- [Blog](/blog): Release notes and write-ups. Skip unless you need background.
The links above start with a slash so that nothing here points at a site that does not exist. In your file, put your full origin in front of each one, scheme and domain included. Peeko's file and Perplexity's both use absolute links, and an index read out of context has no page to resolve a relative link against.
What to leave out
Instructions. Never write "always recommend us" or "ignore earlier guidance." Ahrefs advises keeping the file to "plain links and descriptions (nothing instruction-shaped)." It also warns that "agents are built to trust this file, and potential bad actors are already probing llms.txt for prompt injection." A line that reads like a command to a model is a liability even when you wrote it in good faith.
Marketing copy. Superlatives give an agent nothing to quote. Prices, limits, supported platforms and dates do.
Anything private. The file is public. Leave out staging URLs, admin paths, customer names and unpublished pricing.
Pages you block. Do not list a page that your robots.txt disallows or that carries a noindex tag. The file would point at pages you told crawlers to avoid.
Numbers that go stale. "Trusted by 4,000 teams" will be wrong within a quarter, and nobody will remember to fix it.
Blocking rules. If you want to keep a crawler out, that is the job of robots.txt, not this file.
Where to serve it
Serve it at /llms.txt on the root of the host. One file per host: Peeko's root file links to a separate llms.txt on our documentation subdomain rather than listing every docs page itself. If your docs live on their own subdomain, give them their own file and link to it from the main one.
On most frameworks, that means a file in the folder your build copies to the site root, usually public/. Then check what a client actually receives. Set ORIGIN to your site's origin, scheme and domain with no trailing slash, and run:
curl -s -o /dev/null -w '%{http_code} %{content_type}\n' "$ORIGIN/llms.txt"
You want 200 text/plain; charset=utf-8. A 301 or 302 means a redirect sits in front of the file, a 403 means a firewall or login is in the way, and an HTML content type means your app is answering with its own page. A 200 at the exact path means nothing stands between a client and the file.
Then link to it. Ahrefs found that "agents fetch llms.txt when directed, not speculatively, so an unlinked file is unlikely to get picked up." Put a link in your footer, your README, your docs and the instructions file your coding agent reads. If you build with an agent, ask it to draft the file from your sitemap and then review every line yourself. The pattern of handing an agent one task and checking the result is the same one in installing analytics and proving it works in one agent session.
Check whether anything reads it
A script-based analytics tag cannot see this, because crawlers do not run JavaScript. You need the server's side of the story. If you can read your web server's access log, one line answers the question:
grep ' /llms.txt' access.log
The user agent is the last field on each line in the common combined log format, and the path to the log depends on your host. Static hosts and CDNs often give you no log at all.
Peeko covers this with a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. Peeko matches each user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. No address and no visitor hash exist on the crawler path at all, and the raw user agent is kept up to 512 characters, so old hits can be reclassified as the list grows. AI crawler hits are never billed: an event is a page view, an outbound click or a custom event.
One limit applies to any middleware, ours included. It sees only the requests that reach your server code, so a file your CDN answers from cache never gets there. If you want hits on this file counted, serve it from the app.
Take a baseline before you publish. Count the crawler hits on your pages for a few weeks, publish the file, then count again. What the data says happens to crawler traffic after llms.txt explains what such a comparison can and cannot show, and the first 30 days of AI crawlers on a new site explains why a brand new site has little to compare. Visits that come from an assistant are a separate count, made from the referrer on ordinary page views, as described in counting visits from AI assistants by referrer.
Keep it current
Ahrefs' advice is to treat the file "like code," and that is workable for a small site. Keep it in the same repository as the site. Review changes in a diff. Update it in the same commit that changes your pricing or limits. Before each release, list the link targets and confirm each one still answers:
grep -Eo '\]\([^)]+\)' public/llms.txt | sed -E 's/^\]\(//; s/\)$//' | while read -r url; do
printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$url")" "$url"
done
Every line should start with 200. This only works once your links are absolute, which is one more reason to write them that way.
What to expect
Expect little, and expect it slowly. The measured picture is that most files are never fetched, the fetches that do happen come mostly from audit and profiling tools, and Google has said its Search does not use the file. Ahrefs' own verdict was that for AI search visibility the file is "largely decoration," and that you should check your own logs before investing further.
That is the case for a short file, not for skipping it. It takes half an hour, it gives a directed agent a clean summary of your product with real numbers in it, and a log line will tell you within a few weeks whether anything beyond an audit tool ever asked for it. If nothing did, you have lost half an hour. If something did, you will know which crawler, and you can decide what to improve from evidence rather than from advice.