Claude-SearchBot: what it does, and what blocking it means
The crawler behind Claude's search results. Blocking it reduces or removes your visibility when Claude searches the web.
| Operator | Anthropic |
|---|---|
| What it does | Search & citations |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | Claude |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block Claude-SearchBot?
Blocking it affects
- Claude's search results and citations
It does not affect
- Anthropic model training
Is Claude-SearchBot the same as ClaudeBot?
If you want Claude to find and cite you, this is the token that governs it, not ClaudeBot, which is training only.
It is the same trap as OpenAI's: a broad block aimed at training also catches the search crawler, and the cost lands on your visibility rather than on the training data you meant to protect.
How do you spot Claude-SearchBot in your logs?
To find Claude-SearchBot in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://claude.com)Anyone can send that User-Agent, so treat it as a claim. Anthropic's published IP ranges are the stronger way to verify the request.
Worth knowing about Claude-SearchBot
Anthropic's documentation lists the robots.txt token but publishes no full User-Agent string. The one above is what has been observed in real traffic, not an official string. Last checked 2 September 2026.
How do you block or allow Claude-SearchBot in robots.txt?
To block Claude-SearchBot:
User-agent: Claude-SearchBot
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: Claude-SearchBot
Allow: /With neither rule, Claude-SearchBot is allowed by default.
Not sure how Claude-SearchBot fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block Claude-SearchBot?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 8.6% disallowed Claude-SearchBot in robots.txt. We read the rules for Claude-SearchBot but did not request pages as it, so how many servers would refuse it is not measured.
Is your site blocking Claude-SearchBot?
A CDN or bot-management rule can still block Claude-SearchBot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about Claude-SearchBot
- ClaudeBot vs Claude-SearchBot: which to block?
Anthropic's crawlers split the same way OpenAI's do, and the legacy tokens in most files do nothing at all. - Is Cloudflare blocking AI crawlers?
A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way. - How to detect GPTBot, ClaudeBot and AI bots
A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list.
Official sources
- Anthropic’s crawler documentation
- Published IP ranges is the only way to verify a request claiming this user agent really came from Anthropic
How is Claude-SearchBot different from Anthropic’s other crawlers?
| Claude-SearchBot | ClaudeBot | Claude-User | |
|---|---|---|---|
| Its job | Search & citations | Model training | Fetches when a user asks |
| Blocking it affects | Claude's search results and citations | Anthropic model training | Claude opening your page when someone asks about it |
| Blocking it leaves alone | Anthropic model training | Claude's search results and citations | Claude's search results; Anthropic model training |
| Obeys robots.txt | Yes, per its operator's docs | Yes, per its operator's docs | Yes, per its operator's docs |
| Verifiable by IP | Yes, ranges are published | Yes, ranges are published | Yes, ranges are published |
| Last verified | 2 September 2026 | 2 September 2026 | 2 September 2026 |
Which other crawlers do the same job as Claude-SearchBot?
- OAI-SearchBot: Search & citations, run by OpenAI
- PerplexityBot: Search & citations, run by Perplexity
- meta-webindexer: Search & citations, run by Meta
- MistralAI-Index: Search & citations, run by Mistral
- DuckAssistBot: Search & citations, run by DuckDuckGo