AI crawlers and bots
Count your AI training crawler hits before the shared opt-out token ships
By Jonas Reed · Nov 5, 2026 · 9 min read
What was announced, and what it is not
Most of this week's coverage ends with the same advice: add a group to robots.txt. That takes a minute. What it does not give you is a way to know whether anything happened afterward.
The token comes from an individual IETF Internet-Draft submitted by Kevin Kelley of Anthropic. The IETF has not endorsed it, and it has no formal standing in the standards process. You write it as a user-agent group and list the paths you do not want used for training. The draft's own example blocks a whole site except a press kit:
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/
Three limits matter for what you will be able to measure:
- The draft says the token "is not a directive for crawling behavior." Crawlers MAY skip resources they cannot use for a disallowed purpose, and that is optional. A crawler you want kept out still needs a rule that names it.
- A
User-agent: *group does not count as a preference under the draft. A site with no group for the token has expressed no preference. - Private agreements between a site and an AI company can override it.
Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, 2025-08-29. Retrieved Nov 2, 2026.
The chart shows why the training row is the one to watch. Cloudflare classified AI crawling across its own customers by purpose for the 12 months ending around July 2025. It classed purpose from operator disclosures and industry sources rather than verifying each bot, and its customers are not the whole web. Your site may not look like that average, which is the reason to count your own.
Why adding the line changes nothing you can see today
The rollout is staggered, and the dates are the part to write on a calendar:
- Anthropic: says implementation has started, with no date. It also says the token complements its existing crawler controls rather than replacing them.
- OpenAI: aims to implement it "in the coming months," with no firm date. GPTBot and its other controls still apply.
- Microsoft: a temporary grounding setting opens in Bing Webmaster Tools on Dec 15, 2026, and only for sites that already use NoArchive or NoCache. Microsoft honors the new preferences as it crawls from Feb 1, 2027.
- Google: "early 2027" in its crawler documentation. Google plans to read a shared-token opt-out as if you had disallowed Google-Extended.
So for the next few months the line is a statement of intent that nobody is reading, or that one company has only started to read. If you add it and then see a quiet week, you have learned nothing. If you add it and see a busy week, you have learned nothing either. Without a number from before, both weeks look the same.
Write the baseline first
You need two things: a count per category, and a count per user agent name inside the AI training category. The category tells you the purpose the crawler claims. The name tells you who, so you can match it to the company that just shipped.
Crawlers do not run JavaScript, so a script tag cannot see them. Peeko counts them on a separate server side path, with the @usepeeko/bot-track middleware: ESM only, Node 20 or newer, zero runtime dependencies, under 3 KB gzipped. Install it with:
npm i @usepeeko/bot-track
The middleware file itself is walked through in our Meta crawler post, which uses the same before and after method for a different company's date. Peeko matches each user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. There is no address and no visitor hash on this path at all, and the raw user agent is kept up to 512 characters so old hits can be reclassified as the list grows.
That last point has a consequence for a baseline: a week you count today can come out slightly different if you re-run it after the list gains an agent. So save the output with its date. Pick two full weeks that end before you touch robots.txt. With today at Nov 2, 2026, that is Oct 19 to Oct 25 and Oct 26 to Nov 1:
npx @usepeeko/cli login
npx @usepeeko/cli bots --site example.com --from 2026-10-19 --to 2026-10-25 --json > baseline-1-oct-19-25.json
npx @usepeeko/cli bots --site example.com --from 2026-10-26 --to 2026-11-01 --json > baseline-2-oct-26-nov-1.json
Replace example.com with your domain. The command speaks the same MCP endpoint an agent does, so a command and a tool call return the same numbers. Drop --json to print a table you can read.
Two baseline weeks are not decoration. The gap between them is your noise. If one week has 31 hits from an agent and the next has 38, a later week of 36 tells you nothing, and a later week of 4 does. On a small site both weeks may be single digits, and then the comparison is weak. Say so in your notes instead of reading a trend into it.
If the middleware goes in today, your baseline starts today. If your host keeps access logs, you can get a rough one for last week from a combined-format log. This counts every request whose user agent contains the name, which is a claim, not an identity:
for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User Claude-SearchBot; do
printf '%s %s\n' "$ua" "$(grep -c "$ua" access.log)"
done
Run it against one week of log, and keep the same week boundaries when you run it again. If a few of those names show up in strange places, our log check for scanners in costume covers what to do before you trust the counts.
Re-read the same counts after each company ships
Do the re-read per company, not once. The four will not ship together, and a single comparison in March 2027 would blur four changes into one.
- Note the date a company says the token is live. Do not rely on the announced date alone, because both Anthropic and OpenAI gave none.
- Add the token group to robots.txt on a day you write down, and change nothing else in the file during the comparison. OpenAI says a robots.txt change can take about 24 hours to reach its search systems, so leave a full week anyway.
- Run the same command over a window of the same length, saved to a new file:
npx @usepeeko/cli bots --site example.com --from 2027-02-01 --to 2027-02-07 --json > after-microsoft-feb-1-7.json
- Compare each agent's week against both baseline weeks. Inside the spread of the two baselines means no visible change.
Then read the three outcomes plainly:
- The training row falls and the other rows hold. That is consistent with a crawler skipping resources, which the draft allows. It is not proof that the token caused it, since crawl volume moves for many reasons.
- Nothing moves. This is also consistent with the draft. The token is about use, not crawling, so a crawler can honor it and still fetch the same pages at the same rate.
- The AI answers or Search indexing rows fall. Check what you actually wrote. Google says its reading of the token as a Google-Extended disallow covers grounding in Gemini Apps, and Microsoft warns that blocking grounding "will significantly reduce your content's visibility in AI-generated experiences." Training and answers are separate purposes, but the products that serve them are not always separated cleanly.
One caution about the answers row. Visits where an assistant fetches a page for a person are user driven, and OpenAI says that for ChatGPT-User "robots.txt rules may not apply." Do not expect that row to react to a training preference.
What a count cannot tell you
A count of hits answers one question: which agents arrived, and how often. It does not answer these:
- Whether a crawler read the token. Peeko files hits by user agent, so a fetch of a page tells you an agent was there, not what it did with your robots.txt. Your access log can show whether an agent requested
/robots.txt, which is closer, and the check is in the log post linked above. - Whether your pages were used for training. A fetch is not a use. Training happens off your server, on a schedule you cannot see, and pages fetched before the token ships are already fetched.
- Anything about Google's training crawl. Google's documentation says Google-Extended "doesn't have a separate HTTP request user agent string." Google's fetches arrive under its existing user agents, so there is no Google-Extended row to count, before or after. For Google, the most you can watch is that your Search indexing count does not drop.
- Who sent the request. A user agent is a claim the client writes about itself. Treat an unusual jump in one name as something to check, not a finding.
- Anything about companies outside the four. Meta, Amazon and Apple are not among them. Meta's communications head voiced support for a common signal but did not say whether Meta would honor it. If you use Meta's crawlers as a test case, that count is a separate baseline.
The draft is also at version 00. Rollout dates can slip, and the syntax can change before anyone treats it as stable. Keep your notes dated.
One test you can run today
If you want a change you can measure now, the way to get one is a rule that names a crawler. OpenAI says disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models," and that each of its settings is independent of the others, so a site can allow OAI-SearchBot and disallow GPTBot. That tests whether a named agent honors a rule it already supports. It is a different experiment from the shared token, so do not run it inside the same comparison window. If you add a GPTBot rule in the middle of your token test, you have changed two things at once.
What to do this week
- Install the middleware, or pull a week from your access log.
- Save two full baseline weeks, with the date you took them.
- Add the token group if you want to state the preference. It costs nothing, and the calendar is the only reason to wait. Write down the day.
- Set reminders for Dec 15, 2026, Feb 1, 2027, and early 2027, and add a check for OpenAI and Anthropic announcements.
- When a company says it is live, run the same command over the same length of window and compare.
AI crawler hits are never billed in Peeko, and page views, events and crawler hits are kept three years by default, so the baseline will still be there when Google ships. The count will not tell you whether the token worked. It will tell you whether anything on your server changed, and that is the only part you can check yourself. If you are new to reading these categories, the first 30 days of AI crawlers on a new site shows what ordinary looks like, and counting visits that arrive from assistants by referrer covers the people who click through instead.