AI crawlers, and what blocking each one costs you
19 robots.txt tokens across 13 operators — one of which, xAI, publishes no token at all — with the consequence of blocking each one spelled out. This is the same registry our scanner reads, so nothing here is a summary of a summary. It is what the tool believes about your site, published.
The pattern behind all three: most operators run two crawlers, one that trains and one that finds. They are usually named similarly enough to be copied into a robots.txt interchangeably, and blocking the wrong one is silent. Nothing emails you when your site leaves an AI assistant’s index.
ChatGPT Search
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Search & citations | Yes, per its operator's docs |
| ChatGPT-User | OpenAI | Fetches when a user asks | May bypass, per its operator's docs |
Claude
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| Claude-SearchBot | Anthropic | Search & citations | Yes, per its operator's docs |
| Claude-User | Anthropic | Fetches when a user asks | Yes, per its operator's docs |
Perplexity
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| PerplexityBot | Perplexity | Search & citations | Generally — with caveats |
| Perplexity-User | Perplexity | Fetches when a user asks | May bypass, per its operator's docs |
Google AI (AI Overviews / Gemini)
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| Google-Extended | Model training | Yes, per its operator's docs | |
| Googlebot | Reference | Yes, per its operator's docs |
Other AI assistants
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| Applebot | Apple | Search & training | Yes, per its operator's docs |
| DuckAssistBot | DuckDuckGo | Search & citations | Yes, per its operator's docs |
| Bingbot | Microsoft | Reference | Yes, per its operator's docs |
AI model training
| Token | Operator | What it does | Obeys robots.txt |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Yes, per its operator's docs |
| ClaudeBot | Anthropic | Model training | Yes, per its operator's docs |
| Applebot-Extended | Apple | Model training | Yes, per its operator's docs |
| meta-externalagent | Meta | Model training | Yes, per its operator's docs |
| Amazonbot | Amazon | Search & training | Yes, per its operator's docs |
| Bytespider | ByteDance | Model training | Widely reported not to |
| CCBot | Common Crawl | Open training corpus | Yes, per its operator's docs |
| MistralAI-Training | Mistral | Model training | Yes, per its operator's docs |
Grok (xAI) — no robots.txt control
xAI publishes no official crawler identity. Directory-listed tokens (GrokBot, xAI-Grok) have never been observed in real traffic; Grok fetches with spoofed browser user agents from datacenter IPs. No robots.txt rule can control it — anyone telling you otherwise is guessing.
Source: stackfox.co, a third-party field study. We re-check it quarterly — xAI may yet publish an official identity, and if it does this page changes.
What robots.txt cannot tell you
Every rule above is a declaration. Whether it is honoured is a separate question, and so is whether something in front of your server is blocking these crawlers regardless of what your file says. A CDN rule, a bot-management default or a WAF challenge will turn a crawler away while your robots.txt still reads like an open door — and robots.txt checkers cannot see that layer, because it only shows up when you actually make the request.
That is the check this site exists for. We fetch your homepage once as an ordinary browser and once per AI crawler user agent, and report where the two disagree. Here is exactly what we request from your server.
Scan your own site
Free, no sign-up, about ten seconds.
We request two things: your robots.txt and your homepage. No other pages, ever.
Provenance
Registry version 2026-08-13, last verified 13 August 2026. Every entry carries a source URL from the operator’s own documentation where one exists, and is re-verified quarterly. Where an operator publishes nothing — ByteDance, Meta’s IP ranges, xAI’s identity — we say so on the page rather than filling the gap with a guess.