AI crawlers and bots
Check that your shared pages are noindex and still crawlable
By Theo Marsh · Nov 3, 2026 · 8 min read
If your site makes share links, report pages, previews or generated outputs, each one is a public URL that a search engine can find if the link is posted anywhere. On Oct 26, 2026, Google added a section called "Only index content meant for Search" to its spam guide. Search Engine Journal reports that it covers pages made for sharing, collaboration or utility, and PPC Land reports that Google named no products and gave no enforcement date.
The guide is written for platforms that host other people's pages. If you run one small site, it is not a rule that applies to you. It is a pattern worth copying, and it takes a few minutes to verify with commands you already have.
What the guide says, and what it leaves open
For pages "created for direct sharing, collaboration, or utility rather than public search discovery," Google's guide says to apply the noindex rule, using a robots meta tag or an X-Robots-Tag HTTP response header. It also lists blocking crawling in robots.txt as a default.
Those two are not interchangeable, and they should not be stacked. Google's robots.txt introduction says the file is "not a mechanism for keeping a web page out of Google." Its page on blocking indexing is more direct: if robots.txt blocks a URL, "the crawler will never see the noindex rule, and the page can still appear in search results." Google's robots.txt page adds that "if other pages point to your page with descriptive text, Google could still index the URL without visiting the page."
So a shared page that is disallowed in robots.txt and also carries a noindex tag has a noindex that nobody reads. PPC Land notes that the new guide does not address how the two methods interact, which is where the mistake comes from. A page you want out of the index has to be crawlable, so the crawler can see the instruction.
Which signal sites use today
The HTTP Archive's Web Almanac measured how often pages carry each signal in 2024. The meta tag is the common route, and the response header is rare. Note that these numbers cover all pages in the crawl, not only shared pages, so they say how people usually implement the signal, not how many do it correctly.
Source: HTTP Archive, Web Almanac 2024: SEO, 2024. Retrieved Oct 31, 2026.
The header is the less common choice, but it is the easier one to check with a one line command and the only one that works for non-HTML files such as PDFs. Google's documentation says a response header "can be used for non-HTML resources, such as PDFs, video files, and image files."
Check 1: the response header
Replace the URL with one of your real share links. The -I flag asks for headers only, and -L follows redirects so you test the page a visitor lands on.
curl -sIL "https://example.com/share/3f9a1c" | grep -iE '^(HTTP|x-robots-tag)'
You want output like this, with a 200 and the noindex header:
HTTP/2 200
x-robots-tag: noindex
If you see a 404, a 302 to a sign in page or no header line at all, the check failed. A 302 to a login page is a different problem: the page is not public, so there is nothing to index, but test a link that works without signing in.
Check 2: the meta tag
If you set noindex in the HTML instead, fetch the body and look for the tag:
curl -sL "https://example.com/share/3f9a1c" | grep -io '<meta[^>]*robots[^>]*>'
The expected line is <meta name="robots" content="noindex">. Two limits apply. curl does not run JavaScript, so a tag your front end adds after load will not show here, even if a browser shows it. And the pattern only matches tags with the word robots inside them, so a tag written as googlebot will not match. Google crawls with a rendering step, but a header is present in the first response, which makes it the simpler thing to verify.
Check 3: robots.txt must not block the path
This is the check people skip. Print the disallow rules and compare them to your share path:
curl -s "https://example.com/robots.txt" | grep -inE '^(user-agent|disallow)'
If a line such as Disallow: /share/ covers your URL, Google cannot fetch the page and never sees the header or tag. Remove that rule. If the URLs were already blocked for a while and some got indexed from outside links, removing the rule is also what lets a crawler return, read noindex and drop them.
One script for many URLs
Save this as check-noindex.sh. It takes one URL and prints the status, the header, the meta tag and any disallow line whose value is a prefix of the path.
#!/usr/bin/env bash
# usage: ./check-noindex.sh https://example.com/share/3f9a1c
url="$1"
origin=$(printf '%s' "$url" | sed -E 's#^(https?://[^/]+).*#\1#')
path=$(printf '%s' "$url" | sed -E 's#^https?://[^/]+##')
status=$(curl -sL -o /dev/null -w '%{http_code}' "$url")
header=$(curl -sIL "$url" | tr -d '\r' | grep -i '^x-robots-tag' | tail -1)
meta=$(curl -sL "$url" | grep -io '<meta[^>]*robots[^>]*>' | head -1)
blocked=$(curl -s "$origin/robots.txt" | tr -d '\r' \
| awk -v p="$path" 'tolower($1)=="disallow:" && $2!="" && index(p,$2)==1 {print $0}')
echo "status: $status"
echo "header: ${header:-none}"
echo "meta: ${meta:-none}"
echo "robots.txt: ${blocked:-no matching disallow}"
Run it over a list of links, one per line:
chmod +x check-noindex.sh
while read -r u; do echo "== $u"; ./check-noindex.sh "$u"; done < share-urls.txt
A pass is a 200, a noindex in the header or the meta line, and "no matching disallow". The script is deliberately simple. It ignores user-agent groups and wildcard patterns in robots.txt, so a clean result means no plain prefix rule matched, not that every crawler is allowed. If your robots.txt uses * or $ patterns, read it by hand as well.
Setting the header
If the first check failed, add the header at the server so every page under the share path gets it without touching templates. In Express:
app.use('/share', (req, res, next) => {
res.set('X-Robots-Tag', 'noindex');
next();
});
In nginx:
location /share/ {
add_header X-Robots-Tag "noindex" always;
proxy_pass http://127.0.0.1:3000;
}
If an AI coding agent generated your share route, ask it to put the header in one place for the whole path rather than in each handler, then rerun the script yourself. Handlers added later are the ones that miss it. For a longer walk through letting an agent query your own numbers, see how to connect Claude Code or Cursor to your site analytics.
A concrete case: Peeko share links
A Peeko share link is a page with no sign in behind a 32 character token. A site can have up to ten of them, and each is served noindex. A share link never shows goals, funnels, individual visitors or the AI crawler tab. That last part is the same idea as the guide: a page that exists so someone can read it by link should show only what you meant to hand over, and should not turn up in a search result.
Notice what that design does not do. It does not hide the page behind a robots.txt rule. The page answers to anyone holding the token, including a crawler that finds the link, and what it tells that crawler is noindex. You can apply the same shape to your own report pages: an unguessable token in the path, a short list of fields the page shows, and noindex in the response.
Noindex is an instruction about search results, not a lock. Anyone with the link can still load the page. If a page holds something private, put it behind a sign in, and do not rely on noindex.
Keep your test requests out of your crawler numbers
If you check your pages with a crawler user agent, as in curl -A "Googlebot", remember that your server sees only the string. Peeko matches a crawler user agent against a curated list of 55 agents from 21 companies and files each recognized hit under AI answers, Search indexing, AI training or Other AI bots. A test request that uses a listed string is therefore filed as a crawler hit, and nothing on that path stores an address or a visitor hash, so it cannot be told apart afterward. Test with plain curl, which has its own user agent, and your crawler counts stay clean. The post on spotting a scanner in costume in your logs shows how to tell real crawler traffic from imitations, and keeping your own visits out of site analytics covers your browser visits.
How long until a change shows up
A header or tag you add does nothing until a crawler fetches the page again. Search engines return on their own schedule, which depends on the site. To see the gap on your own site, time the first Google fetch of a new page, and treat the result as one data point, not a promise. For a brand new share page the cleanest approach is to ship the header with the route from the first request, so there is never an indexable version to remove. If you are watching what crawlers do on a young site, the first 30 days of AI crawler traffic shows what to expect to see.
The short checklist
- Fetch a real share URL with
curl -sILand confirm a 200 and anx-robots-tag: noindexline. - If you use the meta tag, confirm it is in the HTML that curl sees, not added by script.
- Read
robots.txtand make sure no rule covers the share path. - Pick one method per path. For link-shared pages, that method is noindex, and the page stays crawlable.
- Rerun the script after every deploy that adds a route, and keep a short list of your share URL patterns next to it.
None of this depends on how Google chooses to act on its new guidance. The same checks hold a month from now, because the behavior they test comes from Google's long-standing documentation: a crawler that cannot fetch a page cannot read what the page says about itself.