GPTBot: what it does, and what blocking it means

OpenAI's training crawler. Blocking it keeps your pages out of future OpenAI models, and does nothing to whether ChatGPT can cite you.

What happens if you block GPTBot?

Does blocking GPTBot block ChatGPT?

GPTBot is the token people reach for when they want to "block AI", and it is usually the wrong one to worry about. It collects content for training. It is not the crawler behind ChatGPT's search answers.

Blocking it is a legitimate policy choice with no search cost. If you meant to stay out of training data, this is the right token and you can stop here. If you meant to control whether ChatGPT recommends you, you want OAI-SearchBot, and you almost certainly want to allow it.

How do you spot GPTBot in your logs?

To find GPTBot in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Anyone can send that User-Agent, so treat it as a claim. OpenAI's published IP ranges are the stronger way to verify the request.

How do you block or allow GPTBot in robots.txt?

To block GPTBot:

User-agent: GPTBot
Disallow: /

To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:

User-agent: GPTBot
Allow: /

With neither rule, GPTBot is allowed by default.

Not sure how GPTBot fits with the others? Build the whole file, or see which AI crawlers to allow.

How many sites block GPTBot?

In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 14.0% disallowed GPTBot in robots.txt, and 15.0% refused a request identifying as GPTBot. For 9.8%, the refusal came from a server whose robots.txt allowed it: a CDN or bot rule deciding something the site’s own file does not say.

Is your site blocking GPTBot?

A CDN or bot-management rule can still block GPTBot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.

We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.

Guides about GPTBot

• How to block AI crawlers and stay in AI search
  The robots.txt that refuses training and keeps citations, built from the registry, plus what to do about the crawlers that do not read the file.
• Why your site is not showing up in ChatGPT
  The common failure modes, in order, with a check you can run for each one.
• GPTBot vs OAI-SearchBot: which to block?
  OpenAI's two crawlers do opposite jobs. One costs you nothing to block and the other takes you out of ChatGPT's answers.
• How to detect GPTBot, ClaudeBot and AI bots
  A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list.
• Do the top 1,000 sites block AI crawlers?
  What the most visited sites say in robots.txt, what their servers actually do, and how often the two disagree.

Official sources

• OpenAI’s crawler documentation
• Published IP ranges is the only way to verify a request claiming this user agent really came from OpenAI

How is GPTBot different from OpenAI’s other crawlers?

Which other crawlers do the same job as GPTBot?

• ClaudeBot: Model training, run by Anthropic
• Google-Extended: Model training, run by Google
• Applebot-Extended: Model training, run by Apple
• meta-externalagent: Model training, run by Meta
• Bytespider: Model training, run by ByteDance
• CCBot: Open training corpus, run by Common Crawl
• MistralAI-Training: Model training, run by Mistral

← All AI crawlers