Googlebot: what it does, and what blocking it means

Still most of your traffic, and still the crawler behind AI Overviews. Blocking it removes you from Google Search entirely.

What happens if you block Googlebot?

Does Googlebot decide whether you appear in AI Overviews?

Worth stating plainly, because the AI conversation has obscured it: AI Overviews and AI Mode are built on ordinary Google Search crawling. Blocking Googlebot does not merely cost you blue links, it takes you out of Google's AI answers at the same time.

Googlebot renders JavaScript, which the AI crawlers largely do not. Content that only appears after your JavaScript runs can be perfectly visible to Google and AI Overviews while being an empty shell to GPTBot, ClaudeBot and PerplexityBot. Two crawlers, same URL, different pages.

We read your robots.txt policy for Googlebot but never send a Googlebot user agent at your site. Cloudflare ships "Fake Google Bot" firewall rules that match the user agent and then fail the IP check, and we cannot present Google's IPs. Any tool that does fetch-test Googlebot is generating guaranteed false positives.

One thing changed on 15 September 2026 that is worth checking even though it has nothing to do with your robots.txt. Cloudflare now classes Googlebot as a mixed-use crawler, one bot doing both search and AI training, and for the first time its AI blocking settings can apply to it. A site owner who switched on something called "Block AI Bots" to keep ChatGPT out may now be turning Google away on pages that carry ads. We cannot see your dashboard and we will not fetch-test this, so we cannot tell you whether it is happening; we can tell you it is newly possible and that the place to look is Cloudflare's AI Crawl Control, under Search rather than Training.

How do you block or allow Googlebot in robots.txt?

To block Googlebot:

User-agent: Googlebot
Disallow: /

To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:

User-agent: Googlebot
Allow: /

With neither rule, Googlebot is allowed by default.

Not sure how Googlebot fits with the others? Build the whole file, or see which AI crawlers to allow.

How many sites block Googlebot?

In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 1.4% disallowed Googlebot in robots.txt. We read the rules for Googlebot but did not request pages as it, so how many servers would refuse it is not measured.

Is your site blocking Googlebot?

A CDN or bot-management rule can still block Googlebot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Guides about Googlebot

• Is Cloudflare blocking AI crawlers?
  A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way.
• Does Google-Extended block AI Overviews?
  A common robots.txt mix-up, and the controls that actually decide what AI Overviews can quote.

Official sources

• Google’s crawler documentation
• Published IP ranges is the only way to verify a request claiming this user agent really came from Google

How is Googlebot different from Google’s other crawlers?

Which other crawlers do the same job as Googlebot?

• OAI-AdsBot: Reference, run by OpenAI
• meta-externalads: Reference, run by Meta
• facebookexternalhit: Reference, run by Meta
• Bingbot: Reference, run by Microsoft

← All AI crawlers