Googlebot
Still most of your traffic, and still the crawler behind AI Overviews. Blocking it removes you from Google Search entirely.
| Operator | |
|---|---|
| What it does | Reference |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | Google AI (AI Overviews / Gemini) |
| Has a crawler | Yes |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
This is worth stating plainly because the AI conversation has obscured it: AI Overviews and AI Mode are built on ordinary Google Search crawling. Blocking Googlebot does not merely cost you blue links — it takes you out of Google's AI answers at the same time.
Googlebot renders JavaScript, which the AI crawlers largely do not. That asymmetry is worth internalising: content that appears only after your JavaScript runs can be perfectly visible to Google and AI Overviews while being an empty shell to GPTBot, ClaudeBot and PerplexityBot. Two crawlers, same URL, different pages.
We evaluate your robots.txt policy for Googlebot but never send a Googlebot user agent at your site. Cloudflare ships explicit "Fake Google Bot" firewall rules that match the user agent and then fail the IP and reverse-DNS check, and we cannot present Google's IPs. Any tool that does fetch-test Googlebot is generating guaranteed false positives.
The robots.txt rules
To block Googlebot:
User-agent: Googlebot
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: Googlebot
Allow: /If neither rule is present, Googlebot is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn Googlebot away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.
Official sources
- Google’s crawler documentation
- Published IP ranges — the only way to verify a request claiming this user agent really came from Google
Other Google tokens
- Google-Extended — Model training