AI crawlers and bots

What an AI visibility score measures and what your server can count

By Jonas Reed 路 Oct 16, 2026 路 7 min read

What the Rank Math announcement says

Rank Math, a WordPress SEO plugin, says AI visibility will become a standard SEO metric in 2027. The Oct 9, 2026 press release names what it means by the term: brand mentions and citations in AI responses, sentiment within AI answers, and position within AI-generated content. It cites a 2026 Pew Research Center survey in which 42% of U.S. adults use AI chatbots to search for information. Its COO, Uzair Nazeer, is quoted: "Rankings and traffic are not going away, but they are no longer the complete picture of search visibility."

That is all this post takes from the story. The question here is narrower. If a client, a boss or a dashboard asks you for an AI visibility number, which part of it can you check against your own traffic?

Where a visibility score comes from

A mention, a sentiment or a position inside an answer is produced by a tool. It sends a list of prompts to an assistant, reads the replies and records what shows up. That is a sample of replies. It is not a record of what any person was shown, and the prompts are chosen by whoever set the tool up.

That does not make the score wrong. It makes it noisy in a way researchers have measured. A 2026 arXiv paper, Don't Measure Once, ran four campaigns of 8 prompts each against ChatGPT, Gemini, Google AI Mode and Perplexity over 45 to 46 days. From one day to the next, cited sources overlapped by 34% to 42% on average, and mentioned brands by 45% to 59%. The authors recommend at least 7 runs per prompt per day and a rolling window of 2 to 4 weeks before reading a per-brand number.

That study is small, so it tells you how far a single run can move, not how any specific product behaves. It does hand you three questions for any score: how many prompts, how many runs, and over what window.

Two counts that do reach your server

Two things an assistant does leave a trace you own.

The first is an answer fetch. When someone asks a question, the assistant may read a page to answer it. OpenAI documents this for ChatGPT-User: "When users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent." Perplexity says Perplexity-User "might visit a web page to help provide an accurate answer and include a link to the page in its response." An answer fetch says a page was read while an assistant was replying to somebody. The question itself never reaches your server, so nobody can report it.

The second is a referred visit. A person clicks a link in an answer and lands on your site, and the request carries the assistant's hostname as its referrer.

These are separate numbers. Cloudflare publishes a crawl-to-refer ratio. Its Radar glossary builds it from two counts: requests for HTML from a platform's crawlers, and requests for HTML that arrived with that platform's hostname in the Referer header. For July 2025, across Cloudflare's network, it was 38,065.7 for Anthropic, 1,091.4 for OpenAI and 194.8 for Perplexity.

38,065.71,091.4194.8AnthropicOpenAIPerplexity

Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, 2025-08. Retrieved Oct 13, 2026.

Read the chart with its limits. The ratio counts all of a platform's crawlers, training included, not only answer fetches. It covers one month on one network, and your site will differ. Cloudflare also notes that traffic referred by native apps may not include a Referer header, so the ratios may be overstated by an amount it cannot say. The point is not the size of any bar. A fetch count and a visit count measure different events, so keep them as two numbers and never add them.

Counting answer fetches with the middleware

Crawlers do not run JavaScript, so a script tag never sees them. Peeko tracks them on a separate server side path with the @usepeeko/bot-track middleware. It is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. Install it:

npm i @usepeeko/bot-track

In Next.js, call it from middleware.ts:

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { trackBotRequest } from "@usepeeko/bot-track";

const cfg = {
  siteKey: process.env.PEEKO_SITE_KEY!,
  endpoint: "https://usepeeko.com/b",
};

export function middleware(request: NextRequest) {
  trackBotRequest(request, cfg); // fire and forget, swallows its own errors
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Peeko matches the user agent against a curated list of 55 agents from 21 companies and files every recognized hit under one of four categories:

  • AI answers: an assistant fetched the page to answer somebody right now, for example ChatGPT-User or Claude-User.
  • Search indexing: ordinary discovery and indexing, for example Googlebot, Bingbot or OAI-SearchBot.
  • AI training: harvesting for model training, for example GPTBot, ClaudeBot or CCBot.
  • Other AI bots: a company Peeko knows, sending an agent it does not recognize by name.

No address and no visitor hash exist on the crawler path at all. The raw user agent is kept up to 512 characters, so old hits can be reclassified as the list grows. Crawler hits are never billed. To tie an agent to the company behind it without storing addresses, see naming the operator of an AI agent without IP addresses.

The split matters because a combined "AI bot" total is mostly something else. Cloudflare put training at 79% of AI crawling in July 2025, search at 17% and user actions at 3.2%.

79%17%3.2%TrainingSearchUser actions

Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, 2025-08. Retrieved Oct 13, 2026.

That is Cloudflare's network, and a small site's mix will differ. But one total would mostly describe training, which is not an answer being given to anyone.

One more limit: whether robots.txt applies changes what you see. OpenAI's documentation says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores them, since a user requested the fetch. Anthropic says its bots, Claude-User included, respect robots.txt. If you block an agent, expect its count to fall, and check each vendor's page before you read a zero as "nobody asked."

Counting answer fetches from your own logs

You do not need Peeko to see the shape of this. If your server writes nginx or Apache combined-format logs, this counts requests whose user agent names one of the three user-action agents above:

awk -F'"' '$6 ~ /ChatGPT-User|Claude-User|Perplexity-User/ {n++} END {print n+0}' /var/log/nginx/access.log

It counts requests, not pages, and it matches a string the sender chose, so read it as requests that say they are those agents. It also misses any assistant whose agent you did not list.

Counting visits from the assistants

Peeko counts visits from ChatGPT, Claude, Perplexity, Copilot and Gemini from ordinary page views by their referrer, with no middleware involved. The same count from a combined-format log:

awk -F'"' '$4 ~ /^https?:\/\/(chatgpt\.com|claude\.ai|(www\.)?perplexity\.ai|copilot\.microsoft\.com|gemini\.google\.com)\// {n++} END {print n+0}' /var/log/nginx/access.log

Asset requests carry your own page as their referrer, so this counts mostly page loads. Treat the result as a floor. Besides the native app gap Cloudflare describes, an assistant that sends no referrer lands in your direct traffic. Counting visits from AI assistants by referrer covers the hostname matching in detail, including why to match exact hostnames and not substrings. Following an AI referral to a signup without cookies covers what happens after the click.

The score and the two counts side by side

Sampled visibility scoreAnswer fetchReferred visit
What it countsMentions, citations, sentiment and position in replies a tool collectedA page read while an assistant was replying to someoneA page view whose referrer is an assistant
Where it comes fromA tool's list of promptsYour server, matched by user agentYour page views, matched by referrer
You can check it yourselfNoYes, in your logs or with the middlewareYes, in your logs or your analytics
What it cannot tell youWhether any person saw that replyThe question, who asked, or whether the page was shownVisits that arrive with no referrer, and the question

What to say when someone asks for an AI visibility number

  • Ask where it came from. If a tool asked the assistants, it is a sample. Ask how many prompts, how many runs and over what window.
  • Put your own counts next to it, labeled. Answer fetches per week and referred visits per week are numbers you can reproduce from your logs. Keep them separate.
  • Keep a baseline. A new site's counts look different from an established one's, and a new site's first 30 days of AI crawler traffic are a poor guide to a normal week.
  • Do not read a count as demand. One answer fetch is one page read for one reply. The question behind it is unknown.

None of this says a sampled score is wrong. A well-run sample can show something your logs cannot, such as whether your brand appears in replies that never fetch or link to your page. Peeko does not measure mentions, sentiment or position, and it cannot see the question behind either count. What it can give you is the half you can check.