Timing the first Google crawl of a new page on your own site

What Google showed on Oct 2

At Search Central Live Deep Dive Europe in Barcelona on Oct 2, 2026, Google's Gary Illyes showed internal timings for several search processes. The figures reached the public through an attendee recap of the session. These are the four that matter for a page you just published:

• Discovery of a new URL: about 20 hours typical, weeks to never at the slowest.
• Refresh of a known URL: about 30 days typical, weeks to never at the slowest.
• Sitemap processing: about 24 hours typical, up to 14 days or never at the slowest.
• Indexing, end to end: about 1.5 hours typical, months or never at the slowest.

Read them with three limits in mind. First, the report says neither the attendee recap nor Google's event pages give a sample size, a time period or a definition of what counts as typical. Second, the figures are a recap of a live talk, not a Google publication, so a wording or rounding error could sit anywhere between the slide and the page. Third, the slowest column marks several "never" cases with the word quality. That suggests a page Google does not think is worth indexing can wait indefinitely, whatever the technical setup.

Source: Search Engine Journal, Google Shows How Long Crawling, Indexing & Recovery Can Take, 2026-10-04. Retrieved Oct 9, 2026.

Google's own help page on recrawl requests gives a wider range. It says "Crawling can take anywhere from a few days to a few weeks," and that requesting a crawl "does not guarantee that inclusion in search results will happen instantly or even at all." So there are two public statements: a typical of about 20 hours, and a documented range of days to weeks. Both can be true for different sites. Neither tells you what your site will get.

What your server can see, and what it cannot

A new page goes through roughly three stages: Google learns the URL exists, a crawler fetches the page, and the page is indexed. Your server sees exactly one of them, the fetch. It does not see discovery, and it does not see indexing.

That shapes what you can claim. The gap you will measure is the time from publish to the first fetch by a search crawler. A page has to be known before it can be fetched, so the first fetch happens at or after discovery. Your gap is therefore a ceiling on the recap's discovery number, not the same quantity. It also says nothing about the 1.5 hours for indexing.

Crawlers do not run JavaScript, so a script tag never sees them. We track them on a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. Peeko matches the user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. Search indexing is the heading you want here.

The crawler path keeps no address and no visitor hash. It keeps the raw user agent, up to 512 characters, so old hits can be reclassified as the list grows. We covered what that does and does not let you say about a crawler's operator in naming the operator of an AI agent without IP addresses.

Install the crawler path

If your site runs on Next.js, install the package and add a middleware.ts at the project root. This is the same setup we describe in what AI crawlers do on a new site in the first 30 days.

npm i @usepeeko/bot-track

import { NextResponse, type NextRequest } from "next/server";
import { trackBotRequest } from "@usepeeko/bot-track";

const cfg = {
  siteKey: process.env.PEEKO_SITE_KEY!,
  endpoint: "https://usepeeko.com/b",
};

export function middleware(request: NextRequest) {
  trackBotRequest(request, cfg); // fire and forget, swallows its own errors
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Put your site key in the environment (PEEKO_SITE_KEY in .env.local for local work, and in your host's settings for production), then deploy. This works only on a site with the middleware installed. If you only have the script tag, you will see people but no crawler hits, and this method will not work for you. Install it before you publish the page you want to time, not after, or you will miss the first visit.

Time one page

Use a page that is new, with a URL no one has requested before. A rewrite of an old URL measures the 30 day refresh case, not discovery.

1. Confirm the page is live, then write down the time in UTC. Check that it returns 200, then record the moment.

curl -s -o /dev/null -w "%{http_code}\n" https://example.com/your-new-page
date -u +"%Y-%m-%dT%H:%M:%SZ"

Use UTC for everything. Hours-long gaps are easy to get wrong when a dashboard and a laptop disagree about time zones.

2. Write down how you announced the page. Did you add it to the sitemap, link it from an existing page, or ask for indexing in Search Console? Pick one and record it. If you do all three, you cannot say which one Google responded to. Linking from a page that is already crawled often is the cleanest test of the default path, because it needs no manual request. Google also notes that there is a quota for submitting individual URLs, and that requesting a recrawl multiple times for the same URL will not get it crawled any faster.

3. Find the first Search indexing hit on that page. Open the crawler tab, filter to Search indexing, and look for your new path. Hits are filed by agent, so check the agent name on the hit. Search indexing covers more than one search engine, and you want the Google crawler, not Bing's. From a terminal, the same data is available through the command line, which needs no install:

npx @usepeeko/cli login
npx @usepeeko/cli bots --site example.com --range 7d

The 30 day post uses that command to note, for each company, the first appearance and the pages requested. Here you do the same for one path.

4. Subtract. The first hit time minus your publish time is the gap. Here is an illustration with made up times, not a measurement: published at 09:00 UTC, first Search indexing hit at 02:40 UTC the next day. The gap is 17 hours and 40 minutes.

Your own test requests are not search crawlers. A curl from your laptop, or a link preview bot that follows a URL you pasted into chat, will not be filed under Search indexing. If you see a hit before you expect one, read the user agent before you celebrate.

Reading the gap

• Under about 20 hours. Consistent with the recap's typical figure. It is one page, so it does not confirm that figure and it does not contradict it.
• A few days. Inside the range Google's help page documents. Check again in a week, and compare it with a second page.
• Nothing after two weeks. Look at the usual causes first: a noindex tag, a robots.txt rule, a page that nothing links to, or a sitemap that does not list it. If all of those are clean, the recap's own wording applies: some of the slowest cases are labeled quality.

The late signal

The crawler hit tells you a search crawler asked for the page. It does not tell you the page is indexed or that anyone saw it in results. That comes from Search Console, which connects to Peeko read only and whose rows we keep for sixteen months. Google publishes search data about three days late and revises its three most recent days, so a page fetched on Monday may not show up in your Search Console rows until later that week.

This is why we treat the two as different tools. The crawler hit is the early signal, available as soon as the fetch happens. The Search Console row is the late one, and the one you can trust more once it settles. If you refresh Search Console every few hours after publishing, you are watching the slower of the two instruments.

Once pages start getting found, the next question is what the visits do. Counting visits from AI assistants by referrer covers the visits that arrive from ChatGPT, Claude, Perplexity, Copilot and Gemini, which need no middleware.

Limits of this method

• One page is one sample. A single 17 hour gap, or a single 4 day gap, is a data point. To say anything about your site's typical, time five or ten pages and write the gaps in a list.
• It does not test Google's 20 hours. The recap's figure has no stated sample or definition, and your measurement starts from a fetch, not from discovery.
• The user agent can be faked. Google says the best way to verify a request really comes from Googlebot is a reverse DNS lookup on the source IP, or a match against its published IP ranges. The crawler path stores no address, so we cannot do that check. A hit that names Googlebot in its user agent is a claim, not proof.
• The middleware sees only requests that reach your app. If your host serves the page without running your code, that fetch will not be recorded.
• Your announcement method matters. A page linked from a busy page and a page left orphaned are different experiments.

What to do this week

Publish one real page, record the UTC time, note how you announced it, and check the Search indexing hits on its path a day later. Write the gap down next to the date. In a month, when the recap is old news, you will still have your own numbers for your own site, which is the only timing that applies to it.