AI crawlers and bots

Which AI crawlers your robots.txt lets in, and what a rule cannot promise

By Nora Blake · Nov 19, 2026 · 9 min read

You can read your own answer in about two minutes. Start with the file, then read what each rule says to each crawler, then look at the rules that quietly do not do what people think. The last section covers how to see which crawlers actually showed up afterward, because the file only tells you what you asked for.

Start with the file itself

Two commands show you what a crawler sees. The first prints the HTTP status, the second prints the file.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/robots.txt
curl -s https://example.com/robots.txt

Replace example.com with your domain. The status matters as much as the content. Google's documentation says its crawlers treat all 4xx errors, except 429, as if a valid robots.txt file did not exist, which means no restrictions. If the file exists but cannot be fetched because of a 5xx error, Google stops crawling the site for the first 12 hours, then uses the last good cached copy for up to 30 days (Google's robots.txt specification). Those are Google's rules for Googlebot. Other crawlers document their own behavior, or none.

If the second command prints a 404 page or an HTML error, you have no robots.txt. Every AI crawler that checks it will read that as permission.

Most sites have said nothing to AI crawlers

If your file has no AI rules, you are in the majority. Cloudflare looked at the top 10,000 domains on Jun 6, 2025. A robots.txt file was found on 3,816 of them, and 546 of those files, about 14%, had allow or disallow directives targeting AI bots. GPTBot was the most blocked: 312 domains disallowed it, 250 fully and 62 partially (Cloudflare).

54631225062Targeted AI botsDisallowed GPTBotGPTBot fullyGPTBot partially

Source: Cloudflare, From Googlebot to GPTBot: Who's crawling your site in 2025, Jul 1, 2025. Retrieved Nov 16, 2026.

Read that sample with its limits in mind. It covers the largest sites, not small products. It is one day, more than a year old. And it counts files that mention an AI bot, not files that work as intended. A separate Cloudflare post from Jul 1, 2025 put the share of robots.txt files disallowing GPTBot at 7.8%, Google-Extended at 5.6%, and each of ClaudeBot, anthropic-ai, PerplexityBot and Bytespider at under 5% (Cloudflare). Your own file is the only sample that matters for your site.

What a rule says to each crawler

The names look interchangeable and are not. The useful split is by job: training, search, and fetches triggered by a person.

OpenAI documents three agents (OpenAI crawler documentation):

  • GPTBot crawls content that may be used to train generative AI foundation models. Disallowing it indicates your content should not be used for training.
  • OAI-SearchBot surfaces websites in ChatGPT's search features. Sites that opt out of it will not be shown in ChatGPT search answers.
  • ChatGPT-User visits pages when a person asks ChatGPT or a Custom GPT something. OpenAI says that because these actions are initiated by a user, robots.txt rules may not apply, and that it is not used to decide whether content appears in Search.

So Disallow: / for GPTBot says nothing to OAI-SearchBot. You can stay out of training and stay in ChatGPT search, or the reverse.

Perplexity documents two agents (Perplexity bots documentation). PerplexityBot is designed to surface and link websites in Perplexity search results, and it is not used to crawl content for AI foundation models. Perplexity-User supports user actions inside Perplexity, and because a user requested the fetch, Perplexity says this fetcher generally ignores robots.txt rules.

Anthropic documents three bots, each with its own robots.txt name: ClaudeBot for training data collection, Claude-User for fetching pages when a Claude user asks a question, and Claude-SearchBot for indexing content for search results. Anthropic says all three honor robots.txt, including Claude-User (Search Engine Journal, Feb 25, 2026). That is a different position from OpenAI's and Perplexity's on user-triggered fetches.

Google-Extended is not a crawler. Google describes it as a standalone robots.txt product token. It does not affect inclusion in Google Search and is not a ranking signal. It lets you opt out of two uses of content Google already crawls: training future Gemini models, and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no HTTP user agent string of its own, because crawling uses existing Google user agent strings (Google's common crawlers documentation). That has a practical consequence: you will never see a Google-Extended hit in a log, so no log can confirm it is working.

A rule set that keeps training crawlers out and leaves search crawlers in looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Whether you want that split is your call. Blocking a search crawler is what costs you a place in that assistant's answers, so decide it on purpose.

Read your file the way a crawler does

Most wrong robots.txt files are right in the editor and wrong in the parser. Google's documentation lists the rules that cause it:

  • Only one group is valid for a particular crawler, and it is the group with the most specific user agent that matches. A crawler does not combine its group with the * group.
  • Path values are case-sensitive. Disallow: /Private/ does not cover /private/.
  • The file size limit is 500 KiB, and content after that is ignored.
  • Google generally caches the file for up to 24 hours.

The first rule is the one that bites. Take this file:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /drafts/

You may expect GPTBot to be kept out of both folders. Under the parsing Google documents, GPTBot reads only its own group, so it is kept out of /drafts/ and is free to fetch /private/. If you add a group for a named crawler, repeat every line from the * group that should still apply to it.

Run the check without reading the file by hand

The free AI crawler checker takes any website and shows which AI crawlers, like GPTBot, ClaudeBot and PerplexityBot, its robots.txt allows or blocks, with no signup. It reads robots.txt and nothing else, and Googlebot is not on its list. Use it after each edit to the file, and treat its output as a statement of what your file requests, not of what happened.

What a rule cannot promise

This is the part to be precise about, because an overstated promise here means a false sense of control.

It cannot enforce anything. Google's introduction to robots.txt says the instructions cannot enforce crawler behavior. Cloudflare describes compliance, per RFC 9309, as voluntary and not access control, and calls robots.txt an honor system (Cloudflare). If you need a crawler kept out, that takes server or firewall rules.

It does not govern every fetch. OpenAI says its user-initiated fetcher may not follow it. Perplexity says its user-initiated fetcher generally ignores it. A page can be read while someone is asking an assistant about it even if your file says no. The question itself never reaches your server, so nobody can tell you what was asked.

It does not keep a page out of search. Google says robots.txt is not a mechanism for keeping a web page out of Google, and that a disallowed page can still be indexed if other sites link to it. For that, Google points to noindex or password protection. If you use shared or private pages, check that your shared pages are noindex and still crawlable.

It is not instant. OpenAI says it can take about 24 hours from a robots.txt update for its search systems to adjust. Perplexity says up to 24 hours. Plan to check the effect a day after an edit, not an hour after.

It does not prove who is asking. A user agent string is text a client chooses to send. Perplexity itself advises combining user agent matching with IP address verification from its published lists. You can see that a firewall rule keyed on the name alone is weak with one command:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://example.com/

That request is not GPTBot. It is your terminal saying so. If it gets a different answer than a plain curl, you have a rule that keys on the name, and anyone can send that name. The same fact limits any log: a hit filed under GPTBot is a request that said it was GPTBot.

Check what actually arrived

A file tells you what you asked for. Your server tells you what happened, and crawlers do not run JavaScript, so a script tag cannot see them. That takes a server side path.

Peeko's is the @usepeeko/bot-track middleware. It is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. It matches a crawler user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. There is no address and no visitor hash on this path at all. The raw user agent is kept up to 512 characters, so old hits can be reclassified when the list grows. AI crawler hits are never billed. If you would rather read the numbers from a terminal than a dashboard, this walkthrough shows how.

Those four categories line up with the split in your robots.txt. Training hits should stop or thin out after you disallow a training crawler. Search indexing and AI answers hits are the ones you asked to keep. The test is simple: change one rule, wait the 24 hours the vendors state, and compare the hits for that crawler before and after. If hits for a name you disallowed continue, two explanations are likely and you should rule out the cheaper one first. Either the crawler is ignoring you, or a scanner is using the name. How much of your AI crawler count could be one scanner in costume walks through separating those. To see the baseline before you touch anything, count your AI training crawler hits first, and for a young site, see which AI crawlers reach a new site in its first 30 days.

A small shop front with a sign hung on an unlocked door, while several small robots approach and some walk around to a side window.

A routine you can repeat

  1. Run the two curl commands. Confirm a 200 and a file under 500 KiB.
  2. Decide, per job, what you want: training in or out, search in or out, answer fetches in or out.
  3. Write one group per crawler name, and repeat any shared lines inside each named group.
  4. Run the checker and compare its verdicts to what you intended.
  5. Wait 24 hours, then compare crawler hits for each name against your intent.
  6. For anything that must stay out, add a server or firewall rule. Do not rely on the file.

The file is worth keeping tidy. It is the one public place where you can state your preference to every well behaved crawler at once. Just do not read it as more than that: a public request, checked by the parser of whoever reads it, honored by the ones who choose to.