DeployRadar
Get helpFree scan

Google-Extended: what it does, and what blocking it means

The token people block to hide from AI Overviews. It does not do that, and this is the most expensive misunderstanding in robots.txt today.

OperatorGoogle
What it doesModel training
Obeys robots.txtYes, per its operator's docs
GovernsGoogle AI (AI Overviews / Gemini)
Has a crawlerNo. A robots.txt token only, nothing fetches under this name
Last verified2 September 2026
Next review2 December 2026. We re-check every entry against the operator’s documentation each quarter.

What happens if you block Google-Extended?

Blocks Gemini model training and grounding (content fed from the Search index to Gemini at prompt time). DOES NOT remove you from AI Overviews or AI Mode, and does not affect Search ranking. If you blocked this to hide from AI Overviews, it isn't doing what you think.

Blocking it affects

  • Gemini model training
  • Gemini grounding: Search content fed to Gemini at answer time

It does not affect

  • AI Overviews and AI Mode
  • Your Google Search ranking

Does blocking Google-Extended remove you from AI Overviews?

Google-Extended controls Gemini training and grounding, meaning content fed from the Search index to Gemini at prompt time. It does not remove you from AI Overviews, it does not remove you from AI Mode, and it does not affect your ranking.

AI Overviews and AI Mode ride ordinary Google Search crawling. If you blocked Google-Extended to stay out of them, you gave up Gemini grounding for nothing.

The levers that do govern AI Overviews are snippet and indexing controls: nosnippet, data-nosnippet, max-snippet and noindex. Those are meta tags and directives, a different file and a different layer from the one most people go looking in.

There is also no Google-Extended crawler. It is a robots.txt token only, and the fetching is done by Google's existing user agents. No tool can test it by making a request, ours included, so anything claiming to have checked it at your CDN is describing something that cannot happen.

How do you block or allow Google-Extended in robots.txt?

To block Google-Extended:

User-agent: Google-Extended
Disallow: /

To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:

User-agent: Google-Extended
Allow: /

With neither rule, Google-Extended is allowed by default.

Not sure how Google-Extended fits with the others? Build the whole file, or see which AI crawlers to allow.

How many sites block Google-Extended?

In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 12.2% disallowed Google-Extended in robots.txt. We read the rules for Google-Extended but did not request pages as it, so how many servers would refuse it is not measured.

Is your site blocking Google-Extended?

A CDN or bot-management rule can still block Google-Extended even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Guides about Google-Extended

Official sources

How is Google-Extended different from Google’s other crawlers?

Google-ExtendedGooglebot
Its jobModel trainingReference
Blocking it affectsGemini model training; Gemini grounding: Search content fed to Gemini at answer timeGoogle Search; AI Overviews and AI Mode
Blocking it leaves aloneAI Overviews and AI Mode; Your Google Search rankingNothing else
Obeys robots.txtYes, per its operator's docsYes, per its operator's docs
Verifiable by IPNo, no published rangesYes, ranges are published
Last verified2 September 20262 September 2026

Which other crawlers do the same job as Google-Extended?

← All AI crawlers