DeployRadar
Get helpFree scan

ClaudeBot vs Claude-SearchBot: which to block?

Short answer: ClaudeBot collects training data and Claude-SearchBot decides whether Claude can find and cite you when it searches. Block ClaudeBot if you want out of training, and keep Claude-SearchBot allowed if you want to be in Claude’s answers. Registry verified 19 September 2026.

What’s the difference between ClaudeBot and Claude-SearchBot?

Anthropic documents three tokens, one job each. Claude-User is the odd one out: it fetches a single page when a person asks Claude about it, rather than crawling.

ClaudeBotClaude-SearchBotClaude-User
Its jobModel trainingSearch & citationsFetches when a user asks
Blocking it affectsAnthropic model trainingClaude's search results and citationsClaude opening your page when someone asks about it
Blocking it leaves aloneClaude's search results and citationsAnthropic model trainingClaude's search results; Anthropic model training
Obeys robots.txtYes, per its operator's docsYes, per its operator's docsYes, per its operator's docs
Verifiable by IPYes, ranges are publishedYes, ranges are publishedYes, ranges are published
Last verified2 September 20262 September 20262 September 2026

Does blocking ClaudeBot hide you from Claude?

No. ClaudeBot gathers pages for training. When Claude searches the web to answer a question, it relies on Claude-SearchBot, and a ClaudeBot rule does not reach it. Blocking ClaudeBot is a policy choice with no search cost.

What happens if you block Claude-SearchBot?

Your visibility in Claude’s search results drops or disappears. This is usually an accident rather than a decision: a broad rule aimed at training catches the search crawler too, and the cost lands on the thing the site owner wanted to keep. The Claude-SearchBot page has the details.

Does Claude-User obey robots.txt?

Anthropic documents that it does, which sets it apart from the equivalent fetchers at some other operators. Blocking Claude-User means Claude cannot open your page for somebody who asked about you by name, a much narrower loss than search visibility but a real one for a product people evaluate by asking.

What about anthropic-ai and claude-web?

Both still appear in a great many robots.txt files, usually copied from lists written in 2023. Neither is in Anthropic’s current documentation, so a rule naming them most likely does nothing. If your file blocks those two and nothing else, it is not blocking Anthropic’s crawlers at all.

What should your robots.txt say?

For a site that wants out of training and into Claude’s answers:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

Write it in robots.txt rather than as an IP block. Anthropic discourages blocking by address, because an IP block can stop the crawler reading robots.txt at all, and then it cannot see the rule you wrote for it.

The checker shows what your current file says to all three. OpenAI splits its crawlers the same way, if you are writing rules for both.

Can a correct robots.txt still keep Claude out?

Yes. A CDN or firewall rule can refuse Claude-SearchBot before it reads the file. The scan requests your home page as each crawler and shows where the answers differ.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Crawlers in this guide

Where to go next

All AI crawler guides