Google-Extended
The token people block to hide from AI Overviews. It does not do that — and this is the single most common expensive misunderstanding in robots.txt today.
| Operator | |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | Google AI (AI Overviews / Gemini) |
| Has a crawler | No — robots.txt token only, nothing fetches under this name |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
Google-Extended controls Gemini model training and grounding: content fed from the Search index to Gemini at prompt time. It does not remove you from AI Overviews, it does not remove you from AI Mode, and it does not affect your Search ranking.
AI Overviews and AI Mode ride ordinary Google Search crawling. If you blocked Google-Extended in order to stay out of them, it is not doing what you think, and you have given up Gemini grounding for nothing.
The levers that actually govern AI Overviews are snippet controls and indexing: nosnippet, data-nosnippet, max-snippet and noindex. Those are meta tags and directives, not user-agent rules — a different file and a different layer from the one most people go looking in.
There is also no Google-Extended crawler. It is a robots.txt token only; the fetching is done by Google's existing user agents. That is why no tool can test it by making a request, ours included, and why anything claiming to have "checked whether Google-Extended is blocked at your CDN" is describing something that cannot happen.
The robots.txt rules
To block Google-Extended:
User-agent: Google-Extended
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: Google-Extended
Allow: /If neither rule is present, Google-Extended is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn Google-Extended away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.
Official sources
Other Google tokens
- Googlebot — Reference