AEO glossary

robots.txt

Also called Robots Exclusion Protocol

robots.txt is a plain-text file at the root of a site that tells crawlers, by user-agent token, which URLs they may fetch, as standardized in RFC 9309.

robots.txt is where you say yes or no to each AI crawler. Google sums it up: "A robots.txt file tells search engine crawlers which URLs the crawler can access on your site." RFC 9309 says the file must be named "/robots.txt" (all lowercase) in the top-level path of the service.

The file is made of groups: one or more User-agent lines followed by Allow and Disallow rules. A crawler matches its product token case-insensitively and obeys that group. Only "If no matching group exists" does it fall back to the User-agent: * group. That is why giving a bot its own group can quietly drop your * rules for it.

Three limits matter:

  • It controls crawling, not indexing. Google says it is "not a mechanism for keeping a web page out of Google" and recommends noindex instead.
  • It is not security. RFC 9309: "These rules are not a form of access authorization."
  • What a block means is set by each operator. A Disallow for GPTBot is a training opt-out; a Disallow for OAI-SearchBot removes you from ChatGPT search answers.

Example

robots.txt
Block one training crawler, keep checkout private for everyone else
User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /checkout/

Learn it properly

The course covers this in Make your site reachable by AI. Lesson 1 is free.

Related terms

Sources

  1. RFC 9309: Robots Exclusion Protocol (opens in a new tab) · IETF (RFC Editor) · Published Sep 2022
  2. Introduction to robots.txt (opens in a new tab) · Google Search Central · Updated 10 Dec 2025
  3. Overview of OpenAI Crawlers (opens in a new tab) · OpenAI · Checked 28 Oct 2026

All 30 glossary terms