How to see which AI crawlers reach a new site in its first 30 days

What we can and cannot tell you

You launched a site nobody links to. You want to know whether AI crawlers find it, which ones come first, and how soon. Those are fair questions, and the honest answer to two of them is that nobody has published numbers for a site like yours.

Every public figure we found describes crawling at scale. It counts requests across millions of sites, weighted by volume, and a site with no inbound links is a rounding error in that data. We will not dress those numbers up as a 30 day forecast. We will show what they do tell you, then show how to get the real answer from your own site.

The first useful number is the split by purpose. Cloudflare Radar measured it over 12 months: "80% of AI crawling was for training, compared with 18% for search and just 2% for user actions."

Source: Cloudflare, The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, 2025. Retrieved Oct 6, 2026.

Two limits on that chart. It describes the share of AI crawling requests on Cloudflare's network, so a few very large sites dominate it. And a share of requests says nothing about how many different companies visit a small site. A new site could see one training request and one answer fetch and look nothing like 80/18/2.

Three jobs, and which ones need a person

AI companies do not run one crawler. They run several, and each has a different job. OpenAI documents three:

• OAI-SearchBot is "used to surface websites in search results in ChatGPT's search features."
• GPTBot is used to crawl "content that may be used in training."
• ChatGPT-User is "used for certain user actions in ChatGPT and Custom GPTs," and OpenAI notes that "robots.txt rules may not apply" because a user started the request.

Perplexity splits the work the same way. PerplexityBot is "designed to surface and link websites in search results on Perplexity" and is not used to crawl content for AI foundation models. Perplexity-User supports user actions, and "since a user requested the fetch, this fetcher generally ignores robots.txt rules."

The split matters for a new site because the three jobs have different triggers. A search crawler or a training crawler has to discover your URL on its own. An answer fetcher does not discover anything: someone asked an assistant a question and the assistant went to read a page. Peeko files recognized crawler hits under four headings, AI answers, Search indexing, AI training and Other AI bots, and the headings line up with these jobs.

An answer fetch tells you a page was read while an assistant was replying to somebody. The question itself never reaches your server, so nobody can report it, not Peeko and not you. You see the page and the time, and that is all.

Why a site nobody links to is a special case

Crawlers find new URLs by following links from pages they already know. Google's sitemap documentation says so directly: "Googlebot and other web crawlers crawl the web by accessing URLs found in previously crawled pages." It also lists new sites with few external links as a case where a sitemap helps, because "Googlebot might not discover your pages if no other sites link to them."

That is Google's statement about Googlebot, and it is the only discovery statement we can cite. The OpenAI and Perplexity pages we read describe what each agent is for and how robots.txt applies. The passages we read give no schedule for finding a new domain, and we will not guess one.

What you can do is remove obstacles. Peeko's install documentation lists three optional files that help crawlers: /llms.txt, /sitemap.xml and /robots.txt. Check the last one first, since a blanket block there makes every other step pointless:

curl -s https://example.com/robots.txt

If you add or change a rule for an OpenAI agent, plan for a delay. OpenAI says it "can take ~24 hours from a site's robots.txt update for our systems to adjust" for OAI-SearchBot, and Perplexity gives a similar window of up to 24 hours.

Who is crawling the most, and why it does not predict your site

For a sense of scale, here is how Cloudflare split AI crawler traffic among the named agents in May 2025.

Source: Cloudflare, From Googlebot to GPTBot: Who's crawling your site in 2025, Jul 1, 2025. Retrieved Oct 6, 2026.

GPTBot grew from 5% to 30% of AI crawler traffic over the year, Meta-ExternalAgent entered at 19%, ClaudeBot fell from 27% to 21%, and Bytespider dropped from 42% to 7%. Googlebot's share of the wider crawler mix rose from 30% to 50%.

Three cautions before you read this as an arrival order. The data is from May 2025, so it is a snapshot and the ordering has probably moved. It is a share of requests, not a count of sites visited, so a company with a heavy crawler can top the chart while visiting fewer sites than a lighter one. And it covers a network of established sites, which have links and history that yours does not.

A crawler visiting also does not mean visitors follow. Cloudflare reported that Anthropic "still crawled 38,000 pages for every referred page visit in July 2025." A crawl is not traffic, and the first one is not a milestone for your human audience. When people do start arriving from an assistant, they show up as referrals, and counting visits from ChatGPT, Claude, Perplexity, Copilot and Gemini is a separate job.

What to expect, labeled as reasoning

None of what follows is a measurement. It is how we would read the first 30 days given what the documentation says.

1. Your own tests will show up first. If you paste your new URL into ChatGPT or Perplexity to see what it says, the assistant may fetch the page with its user action agent. That hit is you. Write down the time of each test so you can subtract it later.
2. Answer fetches follow people, not discovery. They need someone to ask about your page. In week one that someone is usually you or the person you sent the link to.
3. Search and training crawlers follow links. If the only place your URL appears is a social post or a directory listing, expect these to arrive after a link exists that they can follow, not before.
4. Volume will be small. A handful of hits in 30 days is a normal result for an unlinked site, and zero is possible.

If your 30 days disagree with this list, trust your 30 days.

Measure it on your own site

A script tag cannot see a crawler, because crawlers do not run JavaScript. AI crawler tracking in Peeko is a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped.

npm i @usepeeko/bot-track

For Next.js, add this as middleware.ts and set PEEKO_SITE_KEY in your environment:

import { NextResponse, type NextRequest } from "next/server";
import { trackBotRequest } from "@usepeeko/bot-track";

const cfg = {
  siteKey: process.env.PEEKO_SITE_KEY!,
  endpoint: "https://usepeeko.com/b",
};

export function middleware(request: NextRequest) {
  trackBotRequest(request, cfg); // fire and forget, swallows its own errors
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Then read the result from a terminal. Nothing needs installing, and you sign in once:

npx @usepeeko/cli login
npx @usepeeko/cli bots --site example.com --range 30d
npx @usepeeko/cli bots --site example.com --range 30d --provider openai

If you work with an AI coding agent, the same numbers are available to it through the MCP server at usepeeko.com/api/mcp, so you can ask it which companies reached the site this week.

Peeko matches each request's user agent against a curated list of 55 agents from 21 companies. Three properties of that path affect how you read the result:

• It stores no address and no visitor hash. Nothing identifies who ran a crawler, and a hit is classified by its user agent alone.
• It keeps the raw user agent, up to 512 characters, so old hits can be reclassified as the list grows. A string we do not recognize today can become a named agent later without losing the hit.
• Crawler hits are never billed. An event is a page view, an outbound click or a custom event, so crawler traffic does not count toward your monthly limit.

Crawler hits, like page views and events, are kept three years by default, and you can set that window anywhere from 30 days to ten years.

Reading day 30

When the 30 days are up, pull the list and write down four things for each company:

• the date of its first hit, with your own test fetches removed
• the category, one of AI answers, Search indexing, AI training or Other AI bots
• the pages it asked for, and whether it asked for /robots.txt or /sitemap.xml first
• whether it came back, and how many days later

The first row is your answer to "which came first and how soon." The category column is your answer to "found at all": a training or search hit with no answer fetches means a crawler discovered you, while only answer fetches mean someone brought the URL to an assistant.

Keep the limits of your own data in view. One site and 30 days is a sample of one, and a quiet month does not mean your site is blocked or ignored. It can simply mean nothing linked to you yet. If you see no hits, recheck robots.txt and the middleware, then add one real link from a page that already gets crawled, and run the same command again in two weeks.

Public numbers will keep telling you what the web looks like. Only your own log will tell you what happened to your site.