AI crawlers and bots
How much of your AI crawler count could be one scanner in costume
By Emily Parker · Oct 31, 2026 · 9 min read
What Cequence reported
Cequence Security's researchers describe a single actor that posed as about 32 different AI bots, including ChatGPT, Claude and Perplexity. Their write-up says that on one day in October the actor reached 48% of the Cequence-protected organizations that saw any AI bot traffic. At several organizations, the impersonator accounted for more than 80% of the traffic labeled as AI bot traffic.
The scanner did not read pages. It worked from a list of roughly 90 file paths, the kind that commonly hold secrets: .env configuration files, cloud credential files and private SSH keys. Each address sent a burst of 50 to 190 requests in about eight seconds, then moved on. The same machine claimed to be OpenAI, Anthropic, Meta and Perplexity within seconds of each other, and its fake Claude identities carried the contact address of Anthropic's crawler.
Hackread's coverage adds two details. Nearly all of the scanner's requests were blocked or failed during the observation period. And Fastly reported a separate sample on Oct 13: about 6.3 million requests in 24 hours impersonating AI crawlers and fetchers, from nearly 10,200 addresses, aimed at .env files and Git metadata.
Read the limits before you read the percentages. The 80% figure comes from large organizations that Cequence protects: financial services, retail, telecommunications, energy and software. A small site with a few hundred bot hits a month may look nothing like that. We are not claiming that any share of your AI bot hits is fake. We are saying you cannot tell from a user agent alone, and that you can check.
Why a user agent count cannot settle it
A user agent is a header. The client chooses it, and nothing in the request proves it. Counting crawlers by user agent is still the normal method, and it is a fine one for the honest majority. It just counts what requests say about themselves.
Cloudflare's published AI crawler shares are built this way: its post describes matching crawler names with user-agent strings in HTTP requests. The chart below plots those shares for May 2025. Treat them as shares of claimed identities. The post does not say it checked each string against the vendor's network, and a scanner using the same strings would land in the same bars.
Source: Cloudflare, From Googlebot to GPTBot: who's crawling your site in 2025, 2025-07-01. Retrieved Oct 28, 2026.
The mismatch also runs the other way. Cloudflare reported that when Perplexity's declared crawler was blocked, a crawler it attributed to Perplexity switched to a generic browser user agent meant to impersonate Chrome on macOS, and used addresses outside Perplexity's published range. A real crawler can hide behind a browser name, and a scanner can hide behind a crawler name. The string alone tells you neither.
How our crawler path works, and where it stops
AI crawlers do not run JavaScript, so Peeko tracks them on a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. It matches a crawler user agent against a curated list of 55 agents from 21 companies, and it files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. AI crawler hits are never billed, so a scanner in costume does not use up your event limit.
Three facts about that path matter for this story.
- No address and no visitor hash exist on it at all. That is a privacy choice, and it has a cost. The check that vendors and security teams recommend is matching the source address against the vendor's published ranges. We cannot do it, because we never hold the address. Cequence recommends verifying AI agents through signed requests or published address ranges, backed by behavior-based bot management. Our path does none of the three.
- The raw user agent is kept, up to 512 characters. That lets us reclassify old hits as the crawler list grows. It does not unmask a costume. A scanner that sends a real vendor string is filed under that vendor today and will be filed there after any reclassification, because the string is the only evidence we hold.
- The tab tells you which names are claimed, how often, and under which category. It does not tell you whether the request came from the vendor's network. This post also makes no claim about request paths. The path check below uses your own logs.
So a number in the AI training row means "this many requests said they were an AI training crawler." For almost every site that is close to the truth. The check in the next section tells you whether it is true for yours.
One related count works differently. Visits that arrive from ChatGPT, Claude, Perplexity, Copilot and Gemini are counted from ordinary page views by their referrer, with no middleware involved. That path starts from a page view in a browser, not from a header on a bare request. Cequence says the scanner ignored web pages. We make no claim beyond that, but it is a different measurement, and counting visits from AI assistants by referrer explains how it works.
The check to run on your own logs
Use the tab to pick the agent names with the most hits, then test those names against your raw access log. The commands below assume the common combined log format used by nginx and Apache. Replace access.log with your file. If your host gives you no raw log, ask the same four questions of whatever request log it offers.
1. Does an AI user agent ask for secret paths? This lists the user agents that requested a dotfile or key path while naming an AI crawler:
grep -E 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Meta-ExternalAgent|Amazonbot' access.log \
| grep -E '"(GET|HEAD) /(\.env|\.git|\.aws|\.ssh|\.npmrc|id_rsa|wp-config)' \
| awk -F'"' '{print $6}' | sort | uniq -c | sort -rn
A real crawler follows links and sitemaps. It has no reason to ask for /.env or /.aws/credentials. Cequence advises watching for exactly this mismatch, such as an AI assistant requesting configuration or credential files. Any line of output here is a costume.
2. What did your server answer? This counts the status codes returned for those paths, for any user agent:
grep -E '"(GET|HEAD) /(\.env|\.git|\.aws|\.ssh)' access.log | awk '{print $9}' | sort | uniq -c
You want 404 or 403. A 200 means a file was served, and you should treat every secret in it as exposed.
3. Do the same names ever fetch robots.txt? For each crawler, this prints total hits and robots.txt fetches:
for ua in GPTBot OAI-SearchBot ClaudeBot PerplexityBot Meta-ExternalAgent Amazonbot; do
total=$(grep -c "$ua" access.log)
robots=$(grep "$ua" access.log | grep -c '"GET /robots.txt')
echo "$ua total=$total robots=$robots"
done
OpenAI documents robots.txt settings for GPTBot and OAI-SearchBot, and Perplexity documents them for PerplexityBot. A crawler of those names with hundreds of hits and no robots.txt fetch is suspect. Two limits apply. Use a window of at least a week, because crawlers cache robots.txt: OpenAI says it can take about 24 hours for its systems to adjust after a change. And leave the user-triggered fetchers out of this test. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply, and Perplexity says Perplexity-User generally ignores robots.txt. Those two will not fetch it, and that is not a sign of a costume.
4. Are there bursts? This shows the busiest address and second pairs among AI-named requests:
grep -E 'GPTBot|ClaudeBot|PerplexityBot' access.log \
| awk '{print $1, substr($4,2,20)}' | sort | uniq -c | sort -rn | head
Cequence saw 50 to 190 requests from one address in about eight seconds. Real crawlers can burst too, so a burst alone proves nothing. A burst made of secret-file paths is the signal.
The strongest check: published address ranges
If the four questions come back clean, the best remaining test is the one we cannot run for you. OpenAI publishes address ranges at openai.com/gptbot.json, openai.com/searchbot.json and openai.com/chatgpt-user.json. Perplexity publishes perplexitybot.json and perplexity-user.json. Cloudflare says a bot it verifies must declare itself through a cryptographic Web Bot Auth signature, a published address list with a stable user agent, or reverse DNS.
Take one address from your log and test it against the GPTBot file. The file has a creationTime and a prefixes list of address blocks, so this script reads both IPv4 and IPv6 entries:
curl -s https://openai.com/gptbot.json -o gptbot.json
import ipaddress, json, sys
prefixes = json.load(open("gptbot.json"))["prefixes"]
nets = [ipaddress.ip_network(p.get("ipv4Prefix") or p["ipv6Prefix"]) for p in prefixes]
addr = ipaddress.ip_address(sys.argv[1])
print(addr, "is in the published GPTBot ranges" if any(addr in n for n in nets) else "is NOT in the published GPTBot ranges")
python3 -I check_gptbot.py 198.51.100.7
Use a real address from a GPTBot line in your log in place of 198.51.100.7, which is a documentation address. Download the file again before each check, since the ranges change. An address outside the range is not proof of fraud, because a published list can lag. It is a reason to look closer. Cequence also expects address blocklists to go stale within days against this kind of activity, so use ranges to verify a claim, not to hunt for scanners.
If a secret path returned 200
Sites built quickly with an agent often keep a .env file in the project folder, and a static host can serve it if the folder is the web root. Test your own site, with a nonsense path as a control:
for p in .env .git/config .aws/credentials .ssh/id_rsa this-path-does-not-exist-8431; do
printf '%s ' "$p"; curl -s -o /dev/null -w '%{http_code}\n' "https://your-site.example/$p"
done
Replace your-site.example with your own domain, and only test sites you own. Cequence says the scanner first sent random nonsense paths to find sites that answer every request. If your nonsense path returns 200, your host sends every URL to a fallback page, so open the body of the .env response and confirm it is your home page and not a file. If a real file came back, move it out of the public folder, then rotate every key in it. Cequence's advice for the same problem is to keep secrets out of web-accessible files and confirm that staging and admin sites are not exposed to the internet.
What to take from this
The 48% and 80% figures describe large organizations on one vendor's network. They do not describe your site, and neither do we. What carries over is the method. Any tool that counts AI crawlers from a user agent, ours included, is counting claims, and a claim costs a scanner nothing to make.
Our tab is a good answer to "which AI agents say they visit me, and how often." It is not an answer to "were those visits real." For that, run the four log questions on your top agent, then check an address against the vendor's list. If you want the wider picture of what a new site sees from real crawlers, read what AI crawlers do on a new site in the first 30 days. For why a count from a score and a count from your server can differ, see AI visibility scores vs server-side counts. And for how an operator can be named without storing addresses, see naming the operator of an AI agent without IP addresses.