Free tool
AI Crawler Checker
Enter a domain and see, for each major AI crawler, whether your robots.txt lets it in, and the exact rule that decides it. If an AI search crawler is blocked, you get the lines to add.
It checks 15 crawlers and robots.txt tokens from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Meta, grouped by what each operator says it is for: model training, AI search, or fetching a page because a user asked.
How it works
The checker fetches /robots.txt from your domain once, then applies the rules the way the robots.txt standard, RFC 9309, describes: a crawler follows the group that names it (or the * group if none does), and within that group the longest matching path rule wins, with Allow winning a tie.
It checks the home page path (/). Each crawler’s description comes from its operator’s own documentation, linked from its name in the results.
What it does not check: CDN and firewall bot blocking, JavaScript rendering, or whether a crawler actually visits. Lesson 2, The Crawl Docks, covers all of those.
Questions
Does blocking GPTBot keep my site out of ChatGPT search?
No. OpenAI documents separate agents: GPTBot crawls content that may be used to train its models, while OAI-SearchBot decides which sites can appear in ChatGPT's search features. Each follows its own robots.txt group, so you can block one and allow the other.
Does blocking Google-Extended affect Google Search?
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It is a robots.txt token, not a separate crawler: it controls whether content Google crawls may be used to train future Gemini models and for grounding.
What happens if my site has no robots.txt, or it returns an error?
Under the robots.txt standard (RFC 9309), a missing file (any 4xx response) means crawlers may crawl everything. A server error (5xx) or an unreachable file means they should treat the whole site as disallowed until they can read it again.
The checker says a crawler is allowed. Why might it still not reach my pages?
This tool reads robots.txt only. A CDN or web application firewall can turn bots away before robots.txt is ever consulted, and some fetchers acting for a user, such as Perplexity-User, say they generally ignore robots.txt. Check your CDN's bot settings and your server logs too.
Do you keep the domains I check?
No. The checker fetches your robots.txt once to build the report and does not store the domain or the file.
Learn what to do with the results
Lesson 1, “How answer engines choose sources”, is free. See how AI answers get built, and why they are not ten blue links.
Start lesson 1 free