AEO glossary

ClaudeBot

ClaudeBot is Anthropic's crawler that collects web content for training its generative AI models; blocking it in robots.txt excludes your future content from that training.

ClaudeBot is Anthropic's training crawler. Anthropic says that when a site restricts it, this "signals that the site's future materials should be excluded" from its AI model training datasets. Note the word future: the opt-out is described as applying going forward.

Anthropic runs two other bots with different jobs. Claude-SearchBot navigates the web to improve search results, and blocking it may reduce your visibility in user search results. Claude-User fetches pages when users ask Claude questions. Each has its own robots.txt token, so you can block training while staying available to search.

Three practical points from Anthropic's page:

  • It supports the non-standard Crawl-delay extension if you want to slow ClaudeBot down rather than block it.
  • Rules apply per host: "Please do this for every subdomain that you wish to opt out from."
  • Blocking Anthropic's IP addresses instead of using robots.txt "may not work correctly" as an opt-out.

Example

robots.txt
Block Anthropic training, keep Claude search
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Learn it properly

The course covers this in Make your site reachable by AI. Lesson 1 is free.

Related terms

Sources

  1. Does Anthropic crawl data from the web, and how can site owners block the crawler? (opens in a new tab) · Claude Help Center · Updated 7 Apr 2026

All 30 glossary terms