AI crawlers and bots

Count Meta's crawler hits before the Oct 30 terms take effect

By Nora Blake · Oct 25, 2026 · 10 min read

Meta changed the language in its new business terms this month. According to The SOFX Report, Meta removed a sentence that said it "may use any content from the domains and web pages tied to a business account" to develop and train its products. The terms still take effect on Oct 30, 2026. Meta told the outlet that its crawling terms cover only its ads and business products. The AI Insider reports Meta saying its "use of web crawlers remains unchanged."

Both statements can be true and you still would not know what happens on your own server. That is a measurement question, and it only has an answer if you have numbers from before the date.

A shopkeeper marking tally strokes on a wooden post beside a small gate, while a few small robot visitors of different shapes wait on the path, one holding a lantern.

What the reporting says, and what it leaves open

Two points from the SOFX report matter for a small site.

First, the News/Media Alliance read the edit as a limit on use, not on collection. Its spokesperson, Sam Quigley, said that "with it not being expressly ruled out, it's still potentially on the table." Meta says nothing changed. The Alliance says the edit does not stop collection. You do not have to pick a side to act on this.

Second, the report says neither the updated commercial terms nor the self-serve ad terms mention robots.txt. So the terms do not commit Meta to honoring it.

Meta's own crawler documentation adds one more detail. Of the Meta crawlers it lists, one is described this way: "The Meta-ExternalFetcher crawler may bypass robots.txt because it performs fetches that were requested by the user." The same page says facebookexternalhit "might bypass robots.txt when performing security or integrity checks." Both come from Meta's web crawlers page.

That has a practical consequence. If you disallow Meta's agents in robots.txt and still see a Meta-ExternalFetcher hit, that is behavior Meta documents. It is not evidence that the terms changed anything. Only a before and after comparison can separate the two.

Meta is a large share of AI crawler traffic, which is why the baseline is worth taking

Meta's crawler traffic is not a rounding error. Cloudflare's measurement of its own network for May 2025 put Meta-ExternalAgent at 19% of AI crawler requests, third behind GPTBot and ClaudeBot.

30%21%19%11%7.2%GPTBotClaudeBotMeta-ExternalAgentAmazonbotBytespider

Source: Cloudflare Blog, From Googlebot to GPTBot: who's crawling your site in 2025, 2025-07-01. Retrieved Oct 22, 2026.

Read that chart with its limits. It covers Cloudflare customers, not your site. It is from May 2025, so it is more than a year old. It says nothing about Meta's current volume. We use it only to show that Meta's agents were already a meaningful part of crawler traffic before the new terms. Your own numbers are the ones that matter, and a small site can look nothing like Cloudflare's network.

Meta's five tokens, and what Meta says each one is for

Meta's documentation lists five crawlers. Four are relevant to an AI question, and one is a link previewer:

User agent tokenPurpose as Meta states it
meta-externalagent"training foundation AI models or improving products by indexing content directly"
meta-webindexer"improve Meta AI search result quality for users"
meta-externalfetcher"fetches individual links at a user's request"
meta-externalads"improving advertising and other business-related products and services"
facebookexternalhit"crawl the content of an app or website that was shared" on Meta's apps

Meta's page also says that allowing Meta-WebIndexer "helps us cite and link to your content in Meta AI's responses." That is a tradeoff you may want to weigh separately from training.

Two warnings before you use the table. Do not assume how any of these tokens is filed in your own report. Peeko files every recognized crawler hit under AI answers, Search indexing, AI training or Other AI bots, and the category column in your output is what counts. Also, facebookexternalhit is a link previewer, so it may not appear in an AI category at all. Check where it lands instead of guessing.

Step 1: make sure something is recording crawler hits

Crawlers do not run JavaScript, so a script tag on your page will not see them. Crawler tracking is a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. In Next.js, it installs with one command:

npm i @usepeeko/bot-track

The middleware file calls trackBotRequest(request, cfg) with your site key and the endpoint https://usepeeko.com/b. The crawler setup in the 30 day post shows the full file.

Install it today if you have not. It can only count what arrives after it is running, so a site that installs it on Oct 28 has two days of baseline, not seven. Any other logging that records the user agent works too, such as your host's access log or a CDN dashboard. The method below does not depend on Peeko.

What the Peeko path does give you is a privacy shape that fits this question. No address and no visitor hash exist on the crawler path at all. The raw user agent is kept up to 512 characters. That matters here: if Meta renames or adds an agent after Oct 30, the old hits can be reclassified as the list of crawlers grows, and your baseline does not go stale.

Step 2: write the baseline down

Sign in once, then read crawler hits for a fixed window. The command line needs no install, and it speaks the same endpoint an AI agent does, so a command and a tool call return the same numbers.

npx @usepeeko/cli login
npx @usepeeko/cli bots --site example.com --from 2026-10-16 --to 2026-10-22
npx @usepeeko/cli bots --site example.com --from 2026-10-16 --to 2026-10-22 --json > meta-baseline-oct-16-22.json

Replace example.com with your domain. The second command prints the table. The third saves the raw payload to a file, so you keep exactly what was reported on the day you ran it. Commit that file somewhere you will find it later.

Narrow to one company with --provider and the company's name. The CLI refuses unknown values, so start from the plain bots output and use the name as it appears there. If you cannot find Meta in that list, you have learned something about your install before the date, which is the cheapest time to learn it.

Then write down four things from the baseline:

  1. Hits per week under Meta's name, in total and per token.
  2. Category for each token, as your report files it.
  3. Paths. Open the Pages view and list what Meta requested. Note whether it asked for robots.txt, and whether it went to your home page, your posts or only a few files.
  4. Days with zero. On a small site, many days have no Meta hits. That is useful to know.

If your agent is connected through MCP, you can ask it to do this and paste the output into the file. The MCP setup post shows how to connect Claude Code or Cursor.

Step 3: measure your own noise

A single baseline week is not enough. Crawler traffic on a small site is lumpy. One week can have 3 hits and the next 30, with no change on your end.

Take two windows before the date. For example, Oct 16 to Oct 22 and Oct 23 to Oct 29. The gap between those two weeks is a rough measure of normal variation. The gap is not a statistical test, and with two data points it cannot be one. It is a sanity check on how loud your site is.

The numbers below are an illustration, not a measurement from any site. If the two weeks come out at 12 and 19 Meta hits, then a post-date week of 21 is inside your own range and tells you nothing. If they come out at 12 and 14 and the post-date week is 90, that is worth a closer look at the paths.

Then run the same commands for the same length of time after the date:

npx @usepeeko/cli bots --site example.com --from 2026-10-30 --to 2026-11-05
npx @usepeeko/cli bots --site example.com --from 2026-10-30 --to 2026-11-05 --json > meta-after-oct-30-nov-5.json

Repeat for the second and third weeks. A single week will not settle anything.

Step 4: change one thing at a time

Do not edit robots.txt in the middle of this window. If you add a disallow for meta-externalagent on Oct 25, any drop in hits after Oct 30 can come from either the terms or your rule, and you cannot tell which. Meta also asks you to allow up to 24 hours for robots.txt changes to take effect, per its documentation.

If you want to block or allow a Meta agent, decide that after you have the comparison, or take a second baseline after you make the change. Meta's page shows the form for the agents that list an example:

User-agent: meta-externalagent
Allow: /
Disallow: /private/

This post does not recommend blocking or allowing. That is a call about your content and your goals. The count only tells you what happened.

What to compare after Oct 30, 2026

Look at four differences, in this order:

  • Volume. Is the week after outside the range you measured before?
  • New tokens. Does a Meta user agent appear that was not in the baseline? Because the raw string is kept, you can regroup old and new hits under one name.
  • Category shifts. Did hits that were filed under one category move to another? A move can come from Meta changing what an agent says it does, or from Peeko's list changing, so note the date of any list change.
  • Paths. Did Meta start requesting pages it never touched, such as account related paths or pages behind a login redirect? This is the clearest sign that fetch scope changed, if it did.

Also watch for hits on paths your robots.txt disallows. If you see those from meta-externalfetcher, check Meta's note above first. If you see them from meta-externalagent, which Meta's page does not list as a bypassing agent, that is more notable. Even then, you have a log line that says so, not a ruling on what Meta intended.

What this measurement cannot tell you

State the limits before you share any result.

  • A user agent is a claim. Anyone can send a request that says it is meta-externalagent. The counts are hits that say they are Meta, not proof of who sent them. We cover how Peeko treats this in naming the operator of an AI agent without IP addresses.
  • A fetch is not a use. Peeko can see that a page was requested. It cannot see what Meta does with the page afterward. A page fetched by a training agent and a page fetched for a link preview look the same in volume.
  • A count cannot attribute cause. If hits rise after Oct 30, the terms may be the reason, or Meta may have changed its crawl schedule, or your site may have more pages for it to find. A before and after comparison shows that something moved. It does not show why.
  • Business accounts are the stated scope. According to the reporting, the terms concern sites tied to business accounts. If your site is not linked to a Meta business account, you may see no change at all, and that is a result too.
  • Small sites are noisy. If your baseline is a handful of hits a week, the comparison will be weak. That is a fact about your site, not a flaw in the method.

For a wider view of what else is hitting your site, the earlier post on what each Meta crawler fetched before the Oct 30 terms covers the same agents from a log reading angle, and which crawlers request llms.txt and how often shows the same baseline approach on a single file.

Why do this this week

The terms take effect on Oct 30, 2026. Whatever Meta's agents do after that date, you can only describe the change if you have the days before it on file. If you start counting on Oct 31, you have nothing to compare against.

Crawler hits are never billed in Peeko, so a busy week does not change your event count. The whole check is two commands before the date and two after.

Save the files. In a month, when the story is older and the headlines have moved on, you will still be able to answer the question from your own data.