CCBot: what it does, and what blocking it means
Common Crawl's crawler, the open corpus a great many AI labs train on. The highest-leverage single training opt-out there is.
| Operator | Common Crawl |
|---|---|
| What it does | Open training corpus |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block CCBot?
Blocking it affects
- Future Common Crawl snapshots, and the many models trained on them
It does not affect
- Snapshots already taken
- Any assistant's search results
What does blocking CCBot actually reach?
One block here reaches every lab that trains on Common Crawl snapshots, which is a long list. No other single robots.txt rule covers as much ground.
Two limits. It is not retroactive: existing snapshots contain what they contain, and blocking CCBot today affects future ones only. And Common Crawl itself warns about spoofers sending its user agent, a good reminder that a declared user agent is a claim, not an identity.
How do you spot CCBot in your logs?
To find CCBot in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
CCBot/2.0 (https://commoncrawl.org/faq/)Anyone can send that User-Agent, so treat it as a claim. Common Crawl's published IP ranges are the stronger way to verify the request.
How do you block or allow CCBot in robots.txt?
To block CCBot:
User-agent: CCBot
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: CCBot
Allow: /With neither rule, CCBot is allowed by default.
Not sure how CCBot fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block CCBot?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 17.0% disallowed CCBot in robots.txt. We read the rules for CCBot but did not request pages as it, so how many servers would refuse it is not measured.
Is your site blocking CCBot?
A CDN or bot-management rule can still block CCBot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about CCBot
- How to block AI crawlers and stay in AI search
The robots.txt that refuses training and keeps citations, built from the registry, plus what to do about the crawlers that do not read the file. - How to detect GPTBot, ClaudeBot and AI bots
A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list.
Official sources
- Common Crawl’s crawler documentation
- Published IP ranges is the only way to verify a request claiming this user agent really came from Common Crawl
Which other crawlers do the same job as CCBot?
- GPTBot: Model training, run by OpenAI
- ClaudeBot: Model training, run by Anthropic
- Google-Extended: Model training, run by Google
- Applebot-Extended: Model training, run by Apple
- meta-externalagent: Model training, run by Meta
- Bytespider: Model training, run by ByteDance
- MistralAI-Training: Model training, run by Mistral