ClaudeBot: what it does, and what blocking it means
Anthropic's training crawler. Blocking it excludes you from future Claude training, and does not affect whether Claude can find you when it searches.
| Operator | Anthropic |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block ClaudeBot?
Blocking it affects
- Anthropic model training
It does not affect
- Claude's search results and citations
Does blocking ClaudeBot hide you from Claude?
ClaudeBot collects content for training. Claude's search visibility runs on a different token, Claude-SearchBot, and blocking one does not block the other.
Anthropic discourages blocking by IP rather than by robots.txt: an IP block can stop the crawler reading your robots.txt at all, which leaves it unable to see the rule you wrote for it.
Two legacy tokens, anthropic-ai and claude-web, still appear in a great many files. Neither is in Anthropic's current documentation. We parse them and label them legacy. They are most likely inert, and a rule that does nothing is worth knowing about.
How do you spot ClaudeBot in your logs?
To find ClaudeBot in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)Anyone can send that User-Agent, so treat it as a claim. Anthropic's published IP ranges are the stronger way to verify the request.
Worth knowing about ClaudeBot
Anthropic's documentation lists the robots.txt token but publishes no full User-Agent string. The one above is what has been observed in real traffic, not an official string. Last checked 2 September 2026.
How do you block or allow ClaudeBot in robots.txt?
To block ClaudeBot:
User-agent: ClaudeBot
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: ClaudeBot
Allow: /With neither rule, ClaudeBot is allowed by default.
Not sure how ClaudeBot fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block ClaudeBot?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 14.0% disallowed ClaudeBot in robots.txt, and 16.8% refused a request identifying as ClaudeBot. For 10.6%, the refusal came from a server whose robots.txt allowed it: a CDN or bot rule deciding something the site’s own file does not say.
Is your site blocking ClaudeBot?
A CDN or bot-management rule can still block ClaudeBot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about ClaudeBot
- How to block AI crawlers and stay in AI search
The robots.txt that refuses training and keeps citations, built from the registry, plus what to do about the crawlers that do not read the file. - ClaudeBot vs Claude-SearchBot: which to block?
Anthropic's crawlers split the same way OpenAI's do, and the legacy tokens in most files do nothing at all. - How to detect GPTBot, ClaudeBot and AI bots
A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list. - Do the top 1,000 sites block AI crawlers?
What the most visited sites say in robots.txt, what their servers actually do, and how often the two disagree.
Official sources
- Anthropic’s crawler documentation
- Published IP ranges is the only way to verify a request claiming this user agent really came from Anthropic
How is ClaudeBot different from Anthropic’s other crawlers?
| ClaudeBot | Claude-SearchBot | Claude-User | |
|---|---|---|---|
| Its job | Model training | Search & citations | Fetches when a user asks |
| Blocking it affects | Anthropic model training | Claude's search results and citations | Claude opening your page when someone asks about it |
| Blocking it leaves alone | Claude's search results and citations | Anthropic model training | Claude's search results; Anthropic model training |
| Obeys robots.txt | Yes, per its operator's docs | Yes, per its operator's docs | Yes, per its operator's docs |
| Verifiable by IP | Yes, ranges are published | Yes, ranges are published | Yes, ranges are published |
| Last verified | 2 September 2026 | 2 September 2026 | 2 September 2026 |
Which other crawlers do the same job as ClaudeBot?
- GPTBot: Model training, run by OpenAI
- Google-Extended: Model training, run by Google
- Applebot-Extended: Model training, run by Apple
- meta-externalagent: Model training, run by Meta
- Bytespider: Model training, run by ByteDance
- CCBot: Open training corpus, run by Common Crawl
- MistralAI-Training: Model training, run by Mistral