Claude-SearchBot
The crawler behind Claude's search results. Blocking it reduces or removes your visibility when Claude searches the web.
| Operator | Anthropic |
|---|---|
| What it does | Search & citations |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | Claude |
| Has a crawler | Yes |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
If you want Claude to be able to find and cite your site, this is the token that governs it — not ClaudeBot, which is training only.
The pairing is the same trap as OpenAI's: a broad block aimed at training also catches the search crawler, and the cost lands on visibility rather than on the training data you meant to protect.
The robots.txt rules
To block Claude-SearchBot:
User-agent: Claude-SearchBot
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: Claude-SearchBot
Allow: /If neither rule is present, Claude-SearchBot is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn Claude-SearchBot away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.
Official sources
- Anthropic’s crawler documentation
- Published IP ranges — the only way to verify a request claiming this user agent really came from Anthropic
Other Anthropic tokens
- ClaudeBot — Model training
- Claude-User — Fetches when a user asks