AEO glossary
GPTBot
GPTBot is OpenAI's web crawler for model training: disallowing it in robots.txt tells OpenAI your content should not be used to train its generative AI foundation models.
GPTBot is the switch for training, not for ChatGPT search. OpenAI says: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models."
OpenAI runs separate bots for separate jobs, and "Each setting is independent of the others." Appearing in ChatGPT's search answers is controlled by OAI-SearchBot, so a common setup is to disallow GPTBot and allow OAI-SearchBot. OpenAI gives that combination as an example.
In your logs, GPTBot identifies itself with a user-agent string that includes GPTBot/1.4; +https://openai.com/gptbot. A user-agent string is only a claim, so check suspicious traffic against the IP ranges OpenAI publishes for the bot before you trust it.
For a marketer, the decision is a content-licensing one rather than a visibility one. Blocking GPTBot does not, on OpenAI's own description, remove you from ChatGPT search; blocking OAI-SearchBot does.
Example
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Learn it properly
The course covers this in Make your site reachable by AI. Lesson 1 is free.
Related terms
Sources
- Overview of OpenAI Crawlers (opens in a new tab) · OpenAI · Checked 28 Oct 2026
- RFC 9309: Robots Exclusion Protocol (opens in a new tab) · IETF (RFC Editor) · Published Sep 2022