How to detect GPTBot, ClaudeBot and AI bots

Short answer: search your access logs for each crawler’s user agent token, then check the requesting IP against the list its operator publishes. The first step finds requests that claim to be GPTBot or ClaudeBot. Only the second proves they are. Registry verified 19 September 2026.

How do I find AI crawlers in my server logs?

Every crawler that fetches pages names itself in the User-Agent header, and your web server writes that header into its access log. One command counts them. It works on the default log format of nginx, Apache and most hosts that let you download logs:

grep -oiE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Applebot|meta-externalagent|meta-webindexer|meta-externalfetcher|Amazonbot|Bytespider|CCBot|MistralAI-Training|MistralAI-Index|MistralAI-User|DuckAssistBot" /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

The pattern holds 19 tokens, built from our registry, so it covers training, search and user-triggered crawlers from every operator that publishes one. Change the path to wherever your server writes its log. On a managed host the same data is usually under “raw access logs” or a log drain.

Is that request really from GPTBot?

Not necessarily. A user agent is a string anybody can send, and scrapers borrow the well known ones because some sites wave them through. Common Crawl warns about CCBot spoofers by name. The test that settles it is the source IP: the main operators publish the address ranges their crawlers use, and a request from outside them is not theirs, whatever it calls itself.

List the addresses that claimed to be one crawler:

grep -iE "GPTBot" /var/log/nginx/access.log | awk '{print $1}' | sort -u

Then check each one against the operator’s list. Every JSON list in the table below uses the same format, so one shell function covers all of them:

# Usage: verify <ip> <the operator's IP list URL from the table below>
verify() {
  curl -sL "$2" | python3 -c '
import ipaddress, json, sys
ip = ipaddress.ip_address(sys.argv[1])
ranges = [r for p in json.load(sys.stdin)["prefixes"] for r in p.values()]
hit = next((r for r in ranges if ip in ipaddress.ip_network(r)), None)
print(f"{ip} is in {hit}" if hit else f"{ip} is NOT on the list")' "$1"
}

verify 20.171.207.10 https://openai.com/gptbot.json

Google and Bing also document a reverse DNS check for their own crawlers, which reaches the same answer without downloading a list. Either is fine. The user agent alone is not.

Where each operator publishes its IP ranges

A crawler with no published list cannot be verified from outside its operator. Treat its traffic as a claim, and if it matters, rate limit by behaviour rather than by name.

Which AI crawlers will I never see in my logs?

Google-Extended and Applebot-Extended are robots.txt tokens with no crawler behind them. Nothing ever fetches a page under those names: the operator reads your rule for that token and applies it to content its ordinary crawler already collected. Searching your logs for them will always come back empty, and that tells you nothing about whether the rule is working.

xAI (Grok) publishes no crawler identity at all, so there is nothing to search for. The Grok page covers what is known about how it fetches.

What if an AI crawler never shows up?

Absence is a finding too. If your logs show Googlebot every day and no OAI-SearchBot or Claude-SearchBot in a month, the usual reason is not that they lost interest. A CDN or bot rule is turning them away before the request reaches your server, so it never reaches your log either. Cloudflare does this by default on a lot of sites now.

Your logs cannot show a request that was refused upstream. The scan can: it requests your home page as each AI crawler and as a browser, and reports where the answers differ. To hear about it the week it changes rather than months later, run the same check in CI or let us watch it.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Crawlers in this guide

• GPTBot: Model training, run by OpenAI
• OAI-SearchBot: Search & citations, run by OpenAI
• ClaudeBot: Model training, run by Anthropic
• Claude-SearchBot: Search & citations, run by Anthropic
• PerplexityBot: Search & citations, run by Perplexity
• CCBot: Open training corpus, run by Common Crawl

Where to go next

• Is Cloudflare blocking AI crawlers?
  A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way.
• Why your site is not showing up in ChatGPT
  The common failure modes, in order, with a check you can run for each one.
• Which AI crawlers should you allow?
  The tokens have four jobs. Which ones you want depends on what you are trying to achieve, and the costly mistake is blocking the citation crawler when you meant to block the training one.

All AI crawler guides