DeployRadar
Get helpFree scan

How to block AI crawlers and stay in AI search

Short answer: block the training crawlers in robots.txt, leave the search and citation crawlers alone, and use a server rule for the few that do not read the file. The rules below are built from our crawler registry, verified 19 September 2026.

Which AI crawlers should you block?

“AI crawler” covers four different jobs, and only one of them is what most people mean. Training crawlers collect text for future models. Search and citation crawlers fetch pages so an assistant can quote and link you today. Blocking the first group costs no traffic. Blocking the second takes you out of ChatGPT, Claude and Perplexity answers, with no error anywhere to tell you. Which ones to allow walks through the choice by kind of site.

This is not a hypothetical mistake. In our scan of the Tranco top 1,000, OAI-SearchBot was denied in robots.txt on 8.2% of sites. If those sites had been aiming at training alone, that number would be close to zero.

The robots.txt that blocks AI training and keeps AI search

One group can name several user agents, and each crawler obeys the most specific group that names it. So the training tokens share one Disallow, and the citation crawlers get an explicit Allow that keeps them in if somebody later adds a wildcard block:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: meta-webindexer
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /

Some of those tokens, like Google-Extended and Applebot-Extended, have no crawler behind them. The operator reads the rule while crawling under its ordinary name, so the line works even though nothing in your logs will ever carry that token.

To pick token by token instead, the generator writes the file from the same registry. To check a file you already have, paste it into the checker.

Which AI crawlers ignore robots.txt?

Bytespider is the one our registry marks as “widely reported not to”. A Disallow line for it states what you want and controls nothing, so the block has to happen where the request arrives. On nginx:

if ($http_user_agent ~* "(Bytespider)") {
    return 403;
}

A user agent is a claim, so this stops the requests that announce themselves and nothing else. That is still most of them. Detecting AI crawlers in your logs covers telling a real crawler from one borrowing its name.

The user-triggered fetchers are a separate case. When a person pastes your URL into an assistant, several operators say the fetch may not follow robots.txt, because a person asked for that one page. If a page must not be read by anyone who has its link, it needs a login, not a robots.txt rule.

The crawlers you cannot split

Some operators use one token for search and training at once, with no separate training token, so there is no rule that refuses one job and keeps the other:

The rules above leave it allowed, because blocking costs real visibility. Add it to the Disallow group only if keeping out of training is worth that to you. Apple is the counterexample: Applebot does both jobs, but Applebot-Extended, in the block above, takes the training half on its own.

Should you block AI crawlers in Cloudflare instead?

Only if you check what the switch covers. CDN-level AI blocking acts before robots.txt is read, it is not visible in your file, and it tends to reach search and citation crawlers as well as training ones. In our study, 5.6% of sites refused a citation crawler that their own robots.txt allowed. What Cloudflare blocks by default now is worth reading before you rely on it.

Does blocking AI crawlers remove content they already have?

No. A block stops future fetches. Text already collected, or already in an open archive like Common Crawl, stays where it is. That is the reason to decide early, and also the reason not to panic: a rule added today does its job from today.

Check that the block does what you meant

A robots.txt can be correct and a CDN can still disagree with it, in either direction. The scan requests your home page as each AI crawler and shows which ones were refused, and by which layer.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Crawlers in this guide

Where to go next

All AI crawler guides