AI crawlers, and what blocking each one costs you
26 robots.txt tokens across 13 operators, one of which, xAI, publishes no crawler token at all. For each one, we explain what it does and what blocking it costs. This is the same registry our scanner uses.
This page answers “what is this token?”. If you want to know which ones make sense for your site, see the guide by site type.
Already have a robots.txt? The checker tests it against every entry. Starting from scratch? The generator builds a file from the same registry and flags choices that may not mean what they appear to mean.
Most operators run two crawlers
Many operators have separate training and search crawlers. The names are easy to mix up, and blocking the search crawler can quietly remove your site from answers.
- OpenAITrains on youGPTBotBlocked: out of future OpenAI modelsFinds and cites youOAI-SearchBotBlocked: out of ChatGPT answers and citations
- AnthropicTrains on youClaudeBotBlocked: out of future Claude trainingFinds and cites youClaude-SearchBotBlocked: out of Claude's search results
- GoogleTrains on youGoogle-ExtendedBlocked: out of Gemini training and groundingFinds and cites youGooglebotBlocked: out of Search, AI Overviews and AI Mode
- MetaTrains on youmeta-externalagentBlocked: out of Meta AI training dataFinds and cites youmeta-webindexerBlocked: out of Meta AI answers and citations
- AppleTrains on youApplebot-ExtendedBlocked: out of Apple foundation-model trainingFinds and cites youApplebotBlocked: out of Siri, Spotlight and Safari search
- MistralTrains on youMistralAI-TrainingBlocked: out of Mistral model trainingFinds and cites youMistralAI-IndexBlocked: out of Le Chat's search
Two of these pairs have a guide of their own: GPTBot vs OAI-SearchBot and ClaudeBot vs Claude-SearchBot. For Google, see whether Google-Extended blocks AI Overviews.
Every token, by what it governs
What are you trying to do?
Or filter by purpose or operator
ChatGPT Search
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search & citations | Yes, per its operator's docs | |
| ChatGPT-User | OpenAI | Fetches when a user asks | May bypass, per its operator's docs |
Claude
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| Claude-SearchBot | Anthropic | Search & citations | Yes, per its operator's docs | |
| Claude-User | Anthropic | Fetches when a user asks | Yes, per its operator's docs |
Perplexity
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| PerplexityBot | Perplexity | Search & citations | Generally, with caveats | |
| Perplexity-User | Perplexity | Fetches when a user asks | May bypass, per its operator's docs |
Meta AI
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| meta-webindexer | Meta | Search & citations | Yes, per its operator's docs | |
| meta-externalfetcher | Meta | Fetches when a user asks | May bypass, per its operator's docs |
Google AI (AI Overviews / Gemini)
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| Google-Extended | Model training | Yes, per its operator's docs | ||
| Googlebot | Reference | Yes, per its operator's docs |
Other AI assistants
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| OAI-AdsBot | OpenAI | Reference | Yes, per its operator's docs | |
| Applebot | Apple | Search & training | Yes, per its operator's docs | |
| meta-externalads | Meta | Reference | Yes, per its operator's docs | |
| facebookexternalhit | Meta | Reference | Generally, with caveats | |
| MistralAI-Index | Mistral | Search & citations | Yes, per its operator's docs | |
| MistralAI-User | Mistral | Fetches when a user asks | Yes, per its operator's docs | |
| DuckAssistBot | DuckDuckGo | Search & citations | Yes, per its operator's docs | |
| Bingbot | Microsoft | Reference | Yes, per its operator's docs |
AI model training
| Token | Operator | What it does | Obeys robots.txt | Last verified |
|---|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes, per its operator's docs | |
| ClaudeBot | Anthropic | Model training | Yes, per its operator's docs | |
| Applebot-Extended | Apple | Model training | Yes, per its operator's docs | |
| meta-externalagent | Meta | Model training | Yes, per its operator's docs | |
| Amazonbot | Amazon | Search & training | Yes, per its operator's docs | |
| Bytespider | ByteDance | Model training | Widely reported not to | |
| CCBot | Common Crawl | Open training corpus | Yes, per its operator's docs | |
| MistralAI-Training | Mistral | Model training | Yes, per its operator's docs |
Grok (xAI), no robots.txt control
xAI publishes no official crawler identity. Directory-listed tokens (GrokBot, xAI-Grok) have never been observed in real traffic; Grok fetches with spoofed browser user agents from datacenter IPs. No robots.txt rule can control it, and anyone telling you otherwise is guessing.
Source: stackfox.co, a third-party field study. We re-check it quarterly, and xAI may yet publish an official identity, and if it does this page changes.
Why no robots.txt rule reaches Grok, and the tokens that do not work covers the tokens in circulation, where they came from, and what is left to you.
How a crawler decides which rules apply to it
Under the Robots Exclusion Protocol, a crawler uses the most specific group that names it and ignores the others. A crawler with no matching group falls back to User-agent: * and inherits that rule. That is why an explicit Allow: / for a crawler you want is worth writing: it survives a wildcard Disallow added for another reason.
With no matching rule, a crawler is allowed by default. Accidental blocks are usually something that was added, often from a copied snippet. A few tokens have special fallback behaviour, which we note on their individual pages.
What robots.txt cannot tell you
Every rule here is a declaration. A CDN or WAF can still block a crawler even when robots.txt says Allow. You only see that mismatch by making the request.
That is the layer DeployRadar checks. We fetch your homepage as a normal browser and with AI crawler user agents, then show where the responses differ. See exactly what we request.
Scan your own site
Free, no sign-up, about ten seconds.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Provenance
Registry version 2026-09-19, last verified 19 September 2026. Every entry carries a source URL from the operator’s own documentation where one exists, and is re-verified quarterly. Where an operator publishes nothing, as with ByteDance, Meta’s IP ranges and xAI’s identity, we say so rather than filling the gap with a guess.
The same registry is published as JSON at /crawlers.json, one entry per token with its operator, purpose, robots.txt behaviour, sources and verification date, for use in your own linter, log parser or dashboard.