AI crawler checker
See which AI bots your robots.txt lets in, and which it keeps out.
How to use it
- Type your website's address, like example.com, and press Check.
- To test one page, paste its full address. We check your homepage and that page.
- Read the list: every AI bot shows Allowed or Blocked, grouped by what it does.
- To change a verdict, edit the User-agent rules in your robots.txt and check again.
Questions
Does robots.txt actually stop AI crawlers?
No. robots.txt is a public request, not a lock. Crawlers that follow the standard read it and stay out of what it disallows, but nothing forces a bot to obey. To truly keep a bot out, block it at your server or firewall.
What's the difference between training, AI search and user-triggered bots?
Training crawlers collect pages to train AI models. AI search crawlers index pages so an assistant can find and cite them. User-triggered fetchers load a page only when someone asks an assistant about it, and some of their operators say these fetchers may not follow robots.txt, because a person asked for the page.
If I block GPTBot, will my site disappear from ChatGPT search?
No. GPTBot only collects pages for training OpenAI's models. ChatGPT search uses a separate crawler, OAI-SearchBot, so you can block training and still show up in ChatGPT answers by allowing OAI-SearchBot.
Does blocking Google-Extended hurt my Google Search ranking?
No. Google-Extended is not a crawler. It is a token that tells Google whether pages Googlebot already fetched may be used to train and ground Gemini. Google says it has no effect on inclusion or ranking in Google Search.
What if my site has no robots.txt?
If robots.txt is missing and the server answers with a 4xx error such as 404, the standard says crawlers may read everything. If the server fails with a 5xx error, crawlers that follow the standard treat the whole site as off limits until it answers again.
How can I see which AI bots actually visit my site?
robots.txt only tells you what you asked for. Peeko records AI crawler visits on your server, by bot and by purpose, and shows the visits that arrive from AI answers afterward.
See which bots actually visit
robots.txt is a request. Peeko shows the AI crawlers that really fetch your pages, and the AI answers that send people back.
Start with Peeko