AEO glossary

Google-Extended

Google-Extended is a robots.txt product token, not a separate crawler, that controls whether content Google crawls may be used to train future Gemini models and for grounding.

Google-Extended is a control, not a bot. Google says it "doesn't have a separate HTTP request user agent string" and that "the robots.txt user-agent token is used in a control capacity." Crawling is done by Google's existing crawlers, so you will never see Google-Extended in your logs.

What it controls: whether content Google crawls from your site "may be used for training future generations of Gemini models", and its use for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.

What it does not control: Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." For AI features in Search, such as AI Overviews and AI Mode, Google's AI features guide says robots.txt directives for Googlebot are the control for access.

So disallowing Google-Extended is a Gemini training and grounding decision. It is not a way to leave, or to join, AI Overviews. To limit what Search shows from your pages, Google points to snippet controls such as nosnippet and max-snippet.

Example

robots.txt
Opt out of Gemini training and grounding, stay in Google Search
User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /checkout/

Learn it properly

The course covers this in Make your site reachable by AI. Lesson 1 is free.

Sources

  1. Google's common crawlers (opens in a new tab) · Google Crawling Infrastructure · Updated 15 Oct 2026
  2. AI features and your website (opens in a new tab) · Google Search Central · Updated 10 Dec 2025

All 30 glossary terms