ClaudeBot
Anthropic's training crawler. Blocking it excludes you from future Claude model training, and does not affect whether Claude can find you when it searches.
| Operator | Anthropic |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
ClaudeBot collects content for training. Claude's search visibility is a different token — Claude-SearchBot — and blocking one does not block the other.
Anthropic discourages blocking by IP address rather than by robots.txt, because an IP block can stop the crawler reading your robots.txt in the first place, which leaves it unable to see the rule you wrote for it.
Two legacy tokens, anthropic-ai and claude-web, still appear in a great many robots.txt files. They are absent from Anthropic's current documentation. Our scanner parses them and labels them legacy: they are most likely inert, and a rule that does nothing is worth knowing about.
The robots.txt rules
To block ClaudeBot:
User-agent: ClaudeBot
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: ClaudeBot
Allow: /If neither rule is present, ClaudeBot is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn ClaudeBot away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.
Official sources
- Anthropic’s crawler documentation
- Published IP ranges — the only way to verify a request claiming this user agent really came from Anthropic
Other Anthropic tokens
- Claude-SearchBot — Search & citations
- Claude-User — Fetches when a user asks