Do the top 1,000 sites block AI crawlers?
About one in seven of the most visited sites on the web (14.6%) turned away a request identifying as an AI crawler that their own robots.txt allows. We scanned the Tranco top 1,000 on 1 to 2 October 2026 and got a usable result from 499 of them. Every percentage here is out of those 499.
What we found
| 21.2% | deny at least one AI crawler in robots.txt (106 sites). 17.8% name one explicitly rather than catching it in a wildcard rule. |
|---|---|
| 19.8% | refused at least one request identifying as an AI crawler at the server (99 sites). |
| 14.6% | refused an AI crawler that their robots.txt allows (73 sites). Counting only refusals we could confirm were decided by the user agent, it is 12.2% (61 sites). |
| 5.6% | refused a search crawler, OAI-SearchBot or PerplexityBot, that robots.txt allows (28 sites). Those are the crawlers that decide whether ChatGPT and Perplexity can cite a page. |
The third row is the one that matters. A site that blocks GPTBot in robots.txt has made a decision and written it down. A site whose CDN refuses GPTBot while its robots.txt says “come in” usually has not decided anything: a bot rule or a vendor default made the choice, and nobody on the site’s team can see it from the file they maintain.
Which crawlers get refused?
These are the crawlers the scan requests pages as, so we can compare what robots.txt says about each one with what the server did:
| Crawler | Denied in robots.txt | Refused at the server | Refused while robots.txt allows |
|---|---|---|---|
| ClaudeBot | 14.0% | 16.8% | 10.6% |
| GPTBot | 14.0% | 15.0% | 9.8% |
| meta-externalagent | 13.6% | 12.4% | 9.8% |
| PerplexityBot | 11.4% | 12.0% | 6.8% |
| OAI-SearchBot | 8.2% | 10.4% | 6.8% |
Search crawlers are refused less often than training crawlers, but not by much: OAI-SearchBot at 10.4% against GPTBot at 15.0%. If sites were deliberately opting out of training and staying in AI search, the search crawlers’ numbers would be close to zero. Most of these rules are aimed at “AI” in general, not at a job. Why the two are different crawlers.
What robots.txt says about the rest
For the other tokens we read the rules but did not request pages as the crawler, so this column is all there is. Googlebot is included as the baseline: almost nobody blocks it.
| Token | Denied in robots.txt |
|---|---|
| Bytespider | 17.2% |
| CCBot | 17.0% |
| Google-Extended | 12.2% |
| Amazonbot | 10.4% |
| Perplexity-User | 10.2% |
| ChatGPT-User | 9.8% |
| Claude-User | 9.8% |
| DuckAssistBot | 9.4% |
| Claude-SearchBot | 8.6% |
| Applebot-Extended | 8.6% |
| meta-webindexer | 8.4% |
| Googlebot | 1.4% |
Who is doing the refusing?
Where a refusal carried a recognisable bot-management fingerprint, this is whose it was. AWS WAF comes first, ahead of Cloudflare, which is the opposite of what most of the discussion about AI crawler blocking would lead you to expect.
| AWS WAF challenge | 15 sites |
|---|---|
| Cloudflare challenge platform | 6 sites |
| Cloudflare “Attention Required” | 3 sites |
| Generic block page | 2 sites |
| PerimeterX (HUMAN) | 1 site |
| DataDome | 1 site |
Most refusals carried no fingerprint we recognise, so this table describes the minority that did, not the whole market. How to check Cloudflare’s settings if yours is one of them.
How we measured it
For each site the scanner read robots.txt and evaluated it for every token in our crawler registry. It then requested the home page as a browser, as a neutral bot, and as each probed AI crawler, and compared the responses. The full rules are on the methodology page, and it is the same scan anyone can run from the home page.
Of the 1000 domains, 236 could not be scanned, most of them infrastructure domains with no website, and 265 gave a result we did not trust enough to count. They are left out rather than guessed at.
What this does not show
- Our requests come from our own IP address, not OpenAI’s or Anthropic’s. A rule aimed at fake GPTBots would refuse us and still let the real one through from its published ranges. So every finding here is that a site refused a request identifying as the crawler, not that it blocks the crawler.
- Only five crawlers were requested at the server. The robots.txt table covers the others, and a missing server number means we did not test it, not that nobody refuses it.
- Each site is one scan, from one location, at one point in time.
Citing this
You are welcome to use these numbers. Please link to this page, so readers can find the method and the caveats with them:
DeployRadar, "Do the top 1,000 sites block AI crawlers?", October 2026. https://deployradar.dev/guides/ai-crawler-blocking-studyIs your site in the 14.6%?
The scan runs the same checks on one site in about ten seconds, with no account. It shows what your robots.txt allows next to what your server actually did.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Crawlers in this guide
- GPTBot: Model training, run by OpenAI
- ClaudeBot: Model training, run by Anthropic
- OAI-SearchBot: Search & citations, run by OpenAI
- PerplexityBot: Search & citations, run by Perplexity
- meta-externalagent: Model training, run by Meta
Where to go next
- Is Cloudflare blocking AI crawlers?
A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way. - How to detect GPTBot, ClaudeBot and AI bots
A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list. - How to block AI crawlers and stay in AI search
The robots.txt that refuses training and keeps citations, built from the registry, plus what to do about the crawlers that do not read the file.