Claude-User: what it does, and what blocking it means
The fetcher Claude uses when someone asks about your specific page or product. Blocking it means Claude cannot open your site on a user's behalf.
| Operator | Anthropic |
|---|---|
| What it does | Fetches when a user asks |
| Obeys robots.txt | Yes, per its operator's docs |
| Governs | Claude |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block Claude-User?
Blocking it affects
- Claude opening your page when someone asks about it
It does not affect
- Claude's search results
- Anthropic model training
Does Claude-User obey robots.txt?
This runs when a person asks Claude about you by name or by URL. One request, one page, triggered by a human, not a crawl.
Anthropic documents that it honours robots.txt, which is worth noting because the equivalent fetchers at some other operators do not.
How do you spot Claude-User in your logs?
To find Claude-User in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://claude.com)Anyone can send that User-Agent, so treat it as a claim. Anthropic's published IP ranges are the stronger way to verify the request.
Worth knowing about Claude-User
Anthropic's documentation lists the robots.txt token but publishes no full User-Agent string. The one above is what has been observed in real traffic, not an official string. Last checked 2 September 2026.
How do you block or allow Claude-User in robots.txt?
To block Claude-User:
User-agent: Claude-User
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: Claude-User
Allow: /With neither rule, Claude-User is allowed by default.
Not sure how Claude-User fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block Claude-User?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 9.8% disallowed Claude-User in robots.txt. We read the rules for Claude-User but did not request pages as it, so how many servers would refuse it is not measured.
Is your site blocking Claude-User?
A CDN or bot-management rule can still block Claude-User even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about Claude-User
- ClaudeBot vs Claude-SearchBot: which to block?
Anthropic's crawlers split the same way OpenAI's do, and the legacy tokens in most files do nothing at all.
Official sources
- Anthropic’s crawler documentation
- Published IP ranges is the only way to verify a request claiming this user agent really came from Anthropic
How is Claude-User different from Anthropic’s other crawlers?
| Claude-User | ClaudeBot | Claude-SearchBot | |
|---|---|---|---|
| Its job | Fetches when a user asks | Model training | Search & citations |
| Blocking it affects | Claude opening your page when someone asks about it | Anthropic model training | Claude's search results and citations |
| Blocking it leaves alone | Claude's search results; Anthropic model training | Claude's search results and citations | Anthropic model training |
| Obeys robots.txt | Yes, per its operator's docs | Yes, per its operator's docs | Yes, per its operator's docs |
| Verifiable by IP | Yes, ranges are published | Yes, ranges are published | Yes, ranges are published |
| Last verified | 2 September 2026 | 2 September 2026 | 2 September 2026 |
Which other crawlers do the same job as Claude-User?
- ChatGPT-User: Fetches when a user asks, run by OpenAI
- Perplexity-User: Fetches when a user asks, run by Perplexity
- meta-externalfetcher: Fetches when a user asks, run by Meta
- MistralAI-User: Fetches when a user asks, run by Mistral