What your records can say about who runs an AI agent

The Wikimedia claim, and what it does not show

Wikimedia said on Oct 5 that agents it believes OpenAI operates made millions of API requests, crawled millions of pages and sent hundreds of thousands of queries to the Wikidata Query Service. It also said this may have contributed to a partial outage from May 7 to May 11. The story is reported by eWeek, Dataconomy and Startup Fortune, all on Oct 6.

Treat all of it as allegation. The wording is "may have contributed," not "caused." eWeek reports that Wikimedia's own May incident report blamed "aggressive scrapers" without naming an operator, and that OpenAI said it would analyze the activity and share relevant information as that work progresses. Startup Fortune puts the size of the outage at more than half of requests to the service's external endpoint timing out, and six nodes serving stale data for over 20 hours. None of the three pages we opened explains how Wikimedia tied the traffic to OpenAI, and Dataconomy says outright that it does not. We will not guess at the method either.

That gap is the useful part. Your site faces the same question with far less to go on: is an automated agent hitting this site, and who runs it? The rest of this post is about what your own records can honestly answer.

"AI bot" is not one thing

Before you ask who, ask what kind. Cloudflare reported on Aug 29, 2025 that over the previous 12 months, 80% of AI crawling on its network was for training, 18% was for search and 2% was for user actions. That is an old figure from a different network, and your split will not match it. What carries over is the point that the purpose changes what you do next. A training crawler is a decision about robots.txt. A search crawler is a decision about whether you want to be listed. A user action is a person asking an assistant to read your page.

Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, Aug 29, 2025. Retrieved Oct 7, 2026.

Peeko sorts a recognized crawler hit into one of four buckets: AI answers, Search indexing, AI training or Other AI bots. Those are not the same three buckets Cloudflare uses, so do not line the two up number for number. They do the same job, which is to make you say what a hit was for before you decide whether it matters.

Five questions, ranked by how much evidence they need

When a traffic spike shows up, people tend to jump to the last question on this list. Your records answer the first three and cannot answer the last two.

1. Did a named agent request pages, and how often? Yes. The crawler path matches the user agent against a curated list of 55 agents from 21 companies. A match is a countable hit with a date.
2. What is that agent for? Mostly yes. Each recognized hit is filed under one of the four buckets above. For OpenAI you can also read the operator's own description. OpenAI's bot documentation says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features, GPTBot is used to make its generative AI foundation models more useful and safe, and ChatGPT-User is used for certain user actions in ChatGPT and Custom GPTs.
3. Which company does that agent belong to? Yes, as far as the list is right. The operator comes from the agent's name in the list, not from where the request came from.
4. Did this request really come from that company's network? No. This takes an IP address, and the crawler path has none. No address and no visitor hash exist on that path at all.
5. Did the agent cause a slowdown or an outage? No. A count of hits says how often a name appeared. It does not say how heavy each request was or what else was running on your server. That is the same gap in the Wikimedia coverage: a volume and an operator are reported, and "may have contributed" carries the rest.

We built the crawler path to stop at question three on purpose. Holding an address would let us answer question four, and it would also mean holding something we have decided never to hold. A reader of this blog with a ten page site probably does not need an address-level record, and it is easier to keep a promise about data you never collected.

Why a user agent is a claim

A user agent is a header the client writes. Nothing in the request proves the client is who it says it is. That cuts two ways.

First, anyone can send a header that says GPTBot. A named hit is evidence that something called itself that name, not proof that OpenAI sent it. OpenAI publishes address ranges for its three agents precisely so that site owners can check; its documentation lists them at openai.com/searchbot.json, openai.com/gptbot.json and openai.com/chatgpt-user.json.

Second, an agent that sends an ordinary browser string will not be filed as a crawler. Peeko matches against known names. An agent that looks like Chrome to your server looks like Chrome to us. If a headless browser driven by a script visits your pages, it can pass for a person, and nothing on the crawler path will tag it.

There is also a limit on what the operator's rules cover. OpenAI's documentation says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. So a robots.txt that blocks GPTBot does not mean a ChatGPT-User fetch will stop. If you see that name in your records, it says a page was read while an assistant was replying to somebody. The question that person asked never reaches your server, so nobody can report it, including us.

Check your own data

Four steps, in order. Stop when you run out of evidence.

1. Look for a named agent. Open the AI crawler tab for your site and read the agent names, the bucket each is filed under, and the daily count. Share links never show this tab, so open it signed in. AI crawler hits are never billed, so a spike here does not move your bill.

2. Read the raw user agent. Peeko keeps the user agent up to 512 characters. Open the full string, not just the name it was filed under. Check that the version and any other text are what you would expect from that agent.

3. Label what is not named. If you see a burst of traffic that matches no agent, write it down as unattributed. Do not write "OpenAI" because the timing fits a news story. A plain record looks like this:

Oct 3, 2026: 4,200 requests to /docs, user agent reads as Chrome 128,
no crawler match. Unattributed.

Because the raw string is kept, an agent that is unnamed today can be named later. The list of agents grows, and old hits can be reclassified. Date your label so you know when you wrote it.

4. If you need network proof, use a record we do not keep. Your web server or CDN access log may hold the address. If it does, you can compare it against the operator's published ranges. The command below assumes an nginx log in the default combined format, where the address is the first field, plus jq and grepcidr installed. It lists the addresses that sent a GPTBot user agent but sit outside OpenAI's published GPTBot ranges.

curl -s https://openai.com/gptbot.json | jq -r '.prefixes[].ipv4Prefix' > ranges.txt
grep "GPTBot" /var/log/nginx/access.log | awk '{print $1}' | sort -u > seen.txt
grepcidr -v -f ranges.txt seen.txt

We looked at gptbot.json, and its entries use an ipv4Prefix key. If the file you check has IPv6 prefixes too, handle those separately. An empty result means every address you saw falls inside the published ranges. A non-empty result means something used the name without being inside those ranges. Neither result tells you the cause of a slowdown.

What to do with the answer

If the named agent is one you want, leave it alone. If it is a training crawler you do not want, say so in robots.txt, and expect a well behaved operator to honor it. If it is an answer fetch, think of it as a reader that arrived through an assistant, not as a bot you need to block.

If what you have is a spike with no name, do not reach for an operator. Reduce the problem to what you can see: which paths, which hours, how many. Rate limiting by path works without knowing who is on the other end.

For a baseline of what a young site usually sees from named crawlers, read what AI crawlers did to a new site in its first 30 days. For the other half of the picture, the people who arrive from an assistant's answer, read how to count visits from AI assistants by referrer. Those visits are counted from ordinary page views by referrer, with no middleware involved, and they are a separate measurement from a crawler hit.

The short version of the limits

Your records can name an agent, sort it by purpose and date its visits. They cannot confirm the network behind a request, the intent behind it, or the damage it did. Wikimedia has said a lot about who it thinks was behind its outage, and the coverage so far leaves the method open. Hold your own site to a stricter standard than the headline: name what you can show, label the rest unattributed, and change the label only when you have evidence.