User agent

You meet user agents in two places, and they are not the same thing.

• In requests and logs. MDN describes the User-Agent header as "a characteristic string that lets servers and network peers identify" the requesting application. OpenAI's training crawler, for example, sends a long string that includes GPTBot/1.4; +https://openai.com/gptbot.
• In robots.txt. The User-agent: line names a short product token, such as GPTBot. RFC 9309 says crawlers "MUST use case-insensitive matching" to find the group for their token.

Usually the token also appears in the header string. Google-Extended is the exception: Google says it "doesn't have a separate HTTP request user agent string", so you write rules for it but never see it in logs.

A user-agent string is only a claim. Google says its crawlers identify themselves by the header, the source IP and the reverse DNS hostname, and OpenAI publishes IP lists for its bots. Verify before you trust a request that says it is a known AI crawler.

Example

# Header sent by the bot (from OpenAI's docs):
# Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

User-agent: GPTBot
Disallow: /

Related terms

• robots.txt
• GPTBot
• OAI-SearchBot
• ClaudeBot
• PerplexityBot
• Google-Extended

Sources

1. User-Agent header (opens in a new tab) · MDN Web Docs · Updated 28 Oct 2025
2. RFC 9309: Robots Exclusion Protocol (opens in a new tab) · IETF (RFC Editor) · Published Sep 2022
3. Overview of OpenAI Crawlers (opens in a new tab) · OpenAI · Checked 28 Oct 2026
4. Google's common crawlers (opens in a new tab) · Google Crawling Infrastructure · Updated 15 Oct 2026
5. Overview of Google crawlers and fetchers (user agents) (opens in a new tab) · Google Crawling Infrastructure · Updated 12 Jun 2026