Meta's four crawlers and how to see what each one fetched from your site

What the terms say, and what they leave out

Meta's revised Commercial Terms and Self-Serve Ad Terms take effect on Oct 30, 2026. Search Engine Journal reports that they apply to "any business use of Meta's products, from running ads to managing a Facebook Page," and that continued use counts as agreeing. A clause lets Meta use "web scrapers, bots and other automated tools to crawl and collect content" from domains and pages tied to a business's products or services. If you run a Facebook Page, an Instagram business account or an ad account, that can include your site.

The terms name no crawler and, per the same report, say nothing about robots.txt. A Meta spokesperson told Nieman Lab that content owners can still use robots.txt to manage access. Meta's spokesperson Andy Stone also said there is "no change to how Meta uses web crawlers," according to PPC Land.

The wording has already moved once. Unite.AI reports that the Oct 16, 2026 preview let Meta use content from covered domains to "provide, develop, improve, research, train," and that by Oct 17 that sentence was gone and the word "train" no longer appeared in the provision. So whatever the clause says on the day you read this is not settled. Read the current text before Oct 30.

That leaves one thing you can check yourself: what Meta's bots already do on your own server. Meta does publish a list of its crawlers, and the rest of this post works from that list.

What Meta says each crawler is for

Everything in this section comes from Meta's crawler documentation. Where a user agent is quoted, it is the string Meta lists.

For scale, here is how much AI crawler traffic one of these tokens drew on Cloudflare's network in May 2025.

Source: Cloudflare Blog, From Googlebot to GPTBot: who's crawling your site in 2025, 2025-07-01. Retrieved Oct 18, 2026.

The figure is each bot's share of requests from bots on Cloudflare's AI crawler list, for one month, across Cloudflare's customers. It says nothing about your site, and it is 17 months old. Cloudflare's post called Meta-ExternalAgent "making a strong new entry at 19%." Your own log will look different, and that is the reason to read it.

Meta-ExternalAgent: training

Meta says Meta-ExternalAgent "crawls the web for use cases such as training foundation AI models," and also for improving products by indexing content directly. The listed user agent is meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers), with a bare meta-externalagent/1.1 form also listed. This is the token to look at if your concern is training. Meta's page gives robots.txt rules for it and shows an example with User-agent: meta-externalagent.

Meta-WebIndexer: Meta AI search

Meta says "The Meta-WebIndexer crawler navigates the web to improve Meta AI search result quality for users." The listed user agent is meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers). Meta adds: "Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses." This is the one with a possible upside for you, since a citation is a link.

Meta-ExternalFetcher: a fetch on someone's behalf

Meta says "The Meta-ExternalFetcher crawler fetches individual links at a user's request," and that it "may bypass robots.txt because it performs fetches that were requested by the user." The listed user agent is meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers). Treat it like an answer fetch: a page was read because someone asked for it, not because a crawler walked your site. The request that triggered it never reaches your server, so a log shows the read and nothing about the person.

FacebookExternalHit: link previews

Meta says the primary purpose of FacebookExternalHit is to crawl the content of an app or website shared on Meta apps, to collect the title, description and thumbnail. The listed user agents are facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php) and facebookexternalhit/1.1. Meta says it "might bypass robots.txt when performing security or integrity checks." If you have ever pasted a link into Facebook or Instagram and seen a card appear, this is the fetch behind it. Blocking it is a decision about previews, not about training, so think before you do.

A fifth token on the same page

The page we read also lists Meta-ExternalAds, which Meta says "crawls the web for use cases such as improving advertising," with the user agent meta-externalads/1.1 (+/documentation/sharing/webmasters/web-crawlers). It is outside the four named above, but it is on the same page, so the commands below include it. If your log shows it, you now know what Meta says it is.

What robots.txt can and cannot do here

Meta's page says to block a crawler by adding a disallow for it. To opt out of the training token only:

User-agent: meta-externalagent
Disallow: /

To stay in Meta AI search results, you can leave the indexer allowed, which is also the default when you say nothing:

User-agent: meta-webindexer
Allow: /

Three limits apply, all from Meta's own page. Meta asks you to allow up to 24 hours for robots.txt changes to take effect. Meta-ExternalFetcher may bypass robots.txt. FacebookExternalHit might bypass it for security or integrity checks. So a robots.txt file lowers what you should expect from two of the four tokens. Whether a rule was honored is something only your log can show.

Check your server log first

If you run your own server and keep access logs in the common combined format, this counts requests by token. Change the path to your log file.

grep -ioE 'meta-externalagent|meta-webindexer|meta-externalfetcher|meta-externalads|facebookexternalhit' /var/log/nginx/access.log | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

To see which paths one token fetched, most requested first:

awk -F'"' 'tolower($6) ~ /meta-externalagent/ {split($2, r, " "); print r[2]}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

Swap meta-externalagent for any other token. Three things to look for. First, whether /robots.txt appears in the list before the other paths, which means the crawler at least asked. Second, whether a token you disallowed still fetched pages after the 24 hour window. Third, whether the paths are your articles, your product pages or only a handful of shared links, since that tells you which of the four you are really looking at.

The log has a blind spot. If your site sits on a host that does not give you raw access logs, there is no file to search. For that case, use a middleware that records the crawler hit as it arrives.

Check it in Peeko

Crawlers do not run JavaScript, so a tag on the page never sees them. Peeko tracks them on a separate server side path. The @usepeeko/bot-track middleware is ESM only, needs Node 20 or newer, has zero runtime dependencies and weighs under 3 KB gzipped. Install it:

npm i @usepeeko/bot-track

For Next.js, Peeko's docs show this middleware. It fires and forgets, and swallows its own errors:

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
import { trackBotRequest } from "@usepeeko/bot-track";

const cfg = {
  siteKey: process.env.PEEKO_SITE_KEY!,
  endpoint: "https://usepeeko.com/b",
};

export function middleware(request: NextRequest) {
  trackBotRequest(request, cfg); // fire and forget, swallows its own errors
  return NextResponse.next();
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Then read the result from a terminal. The command line needs no install:

npx @usepeeko/cli bots --range 30d

Per Peeko's CLI docs, this prints AI crawler traffic: which bots read the site and what they read. Add --json for the raw payload. Peeko matches a crawler user agent against a curated list of 55 agents from 21 companies and files every recognized hit under AI answers, Search indexing, AI training or Other AI bots. On this path no address and no visitor hash exist at all, and AI crawler hits are never billed.

Here is the part we will not claim. Peeko's published crawler documentation names examples such as GPTBot, ClaudeBot and ChatGPT-User, but it does not name the Meta tokens, and we have not checked the 55 agents against the five above for this post. So we are not telling you which category each Meta token lands in. Run the command and read the result on your own site. The CLI also takes --provider to narrow results to one company, and it refuses values it does not know, so trying --provider meta gives you an answer either way.

One property matters for a story that is still moving. Peeko keeps the raw user agent for each hit, up to 512 characters, so old hits can be reclassified as the crawler list grows. If a Meta token is filed under Other AI bots today and gets its own entry later, the hits you already recorded can be re-sorted. A tally kept in a spreadsheet by hand cannot do that.

What a user agent proves

A user agent is a string the client chooses. Anyone can send meta-externalagent/1.1, and scrapers do borrow the names of big crawlers. The Meta page we read lists no IP ranges and no reverse DNS procedure for checking a request. It says only that you can allow list the user agent strings or IP addresses, and that IP addresses are the more secure of the two, without listing any. So a count by token tells you what requests said they were. It does not tell you who sent them, and it cannot show what Meta later did with a page.

Peeko cannot add that proof. It never stores an address on the crawler path, which is the point of the design, and the cost is that it cannot verify an origin by address. For how to reason about an operator without addresses, see naming the operator of an AI agent without IP addresses. If you keep raw logs, that is where the address check belongs, on your side.

What to do before Oct 30

Decide per purpose, not per company. Meta published separate tokens for training, AI search, user requested fetches and link previews, and the four have different costs if you block them. A short plan:

1. Run the log command above over the last 30 days and write down which tokens appear and how many requests each made.
2. Check /robots.txt for each token you care about, and confirm the file is served as plain text at the root of each domain tied to your Meta accounts.
3. Decide on Meta-ExternalAgent separately from Meta-WebIndexer. One is about training, and the other is the one Meta says helps it cite and link to you.
4. Re-run the count a few days after any robots.txt change, past the 24 hour window, and note whether Meta-ExternalFetcher and FacebookExternalHit appeared anyway.
5. Read the clause as it stands on the day you decide, given that it changed within a day of its preview.

If you are new to what these hits look like when a site is young, AI crawlers on a new site in the first 30 days shows the typical early pattern. For how a fetch differs from a visit that arrives from an assistant, see counting visits from AI assistants by referrer, and for the gap between a vendor's visibility score and a server side count, see AI visibility scores vs server side counts.

The terms are a legal question, and lawyers will argue about them well past Oct 30. The log is a factual one. It tells you in ten minutes which of Meta's four crawlers has been reading your pages, which pages, and whether your robots.txt rules changed anything.