PerplexityBot: what it does, and what blocking it means
The crawler behind Perplexity's answers. Blocking it means your site is not surfaced or linked in Perplexity results.
| Operator | Perplexity |
|---|---|
| What it does | Search & citations |
| Obeys robots.txt | Generally, with caveats |
| Governs | Perplexity |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block PerplexityBot?
Blocking it affects
- Perplexity answers and links
It does not affect
- Model training, which Perplexity says it does not use crawled content for
Does PerplexityBot obey robots.txt?
Perplexity says it does not use crawled content for foundation-model training, so this token is about visibility rather than training data.
Its compliance history is contested and we will not paper over it: Cloudflare documented undeclared stealth crawling in August 2025 and delisted Perplexity from its verified-bots list. We report what your robots.txt declares as intent, not as a guarantee of behaviour. That is the honest reading for any crawler, and a necessary one here.
How do you spot PerplexityBot in your logs?
To find PerplexityBot in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)Anyone can send that User-Agent, so treat it as a claim. Perplexity's published IP ranges are the stronger way to verify the request.
How do you block or allow PerplexityBot in robots.txt?
To block PerplexityBot:
User-agent: PerplexityBot
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: PerplexityBot
Allow: /With neither rule, PerplexityBot is allowed by default.
Not sure how PerplexityBot fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block PerplexityBot?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 11.4% disallowed PerplexityBot in robots.txt, and 12.0% refused a request identifying as PerplexityBot. For 6.8%, the refusal came from a server whose robots.txt allowed it: a CDN or bot rule deciding something the site’s own file does not say.
Is your site blocking PerplexityBot?
A CDN or bot-management rule can still block PerplexityBot even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about PerplexityBot
- Why your site is not showing up in Perplexity
Perplexity's crawler is refused by CDNs more often than most. The causes in order, with the numbers from our top 1,000 scan. - Is Cloudflare blocking AI crawlers?
A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way. - How to detect GPTBot, ClaudeBot and AI bots
A user agent is a claim, not an identity. How to find each AI crawler in your logs and check it against the operator's own IP list. - Do the top 1,000 sites block AI crawlers?
What the most visited sites say in robots.txt, what their servers actually do, and how often the two disagree.
Official sources
- Perplexity’s crawler documentation
- Published IP ranges is the only way to verify a request claiming this user agent really came from Perplexity
How is PerplexityBot different from Perplexity’s other crawlers?
| PerplexityBot | Perplexity-User | |
|---|---|---|
| Its job | Search & citations | Fetches when a user asks |
| Blocking it affects | Perplexity answers and links | Perplexity opening your page when someone asks (it may ignore the rule) |
| Blocking it leaves alone | Model training, which Perplexity says it does not use crawled content for | Perplexity's search results |
| Obeys robots.txt | Generally, with caveats | May bypass, per its operator's docs |
| Verifiable by IP | Yes, ranges are published | Yes, ranges are published |
| Last verified | 2 September 2026 | 2 September 2026 |
Which other crawlers do the same job as PerplexityBot?
- OAI-SearchBot: Search & citations, run by OpenAI
- Claude-SearchBot: Search & citations, run by Anthropic
- meta-webindexer: Search & citations, run by Meta
- MistralAI-Index: Search & citations, run by Mistral
- DuckAssistBot: Search & citations, run by DuckDuckGo