Why your site is not showing up in Perplexity
Perplexity can only cite a page its crawler was allowed to fetch and could read. Below are the reasons it could not, in rough order of how often each one is the cause. Registry verified 19 September 2026.
Which Perplexity crawler decides whether you are cited?
Perplexity runs two, and only one of them builds the index its answers draw on.
| PerplexityBot | Your site will not be surfaced or linked in Perplexity answers. |
|---|---|
| Perplexity-User | Advisory only. Perplexity's own docs state this user-triggered fetcher 'generally ignores robots.txt rules'. |
PerplexityBot is the one that matters. Perplexity says it does not use crawled content to train foundation models, so unlike OpenAI or Anthropic there is no separate training crawler to block instead: refusing PerplexityBot is refusing the citations.
1. Your CDN or firewall turns PerplexityBot away
For Perplexity this comes first. Cloudflare documented undeclared crawling by Perplexity in August 2025 and removed it from its verified-bots list, which means PerplexityBot no longer gets the pass that list exists to give. Bot-management products are built to refuse exactly this kind of request, and many sites have them on without anybody deciding to.
In our scan of the Tranco top 1,000, 12.0% of sites refused a request identifying as PerplexityBot, and 6.8% did so while their own robots.txt allowed it. Those sites have a correct file and are still out, and no robots.txt checker will show them why. See what Cloudflare blocks by default if your site sits behind it.
2. Your robots.txt blocks it, on purpose or not
11.4% of the sites we measured deny PerplexityBot in robots.txt. Some mean to. Many pasted a “block AI bots” list that included it next to the training crawlers, or have a User-agent: * group with a Disallow that PerplexityBot falls under because nothing names it. If you want to be cited, say so explicitly:
User-agent: PerplexityBot
Allow: /A crawler obeys the most specific group that names it, so this group wins over any wildcard rule elsewhere in the file. Paste your file into the checker to see what it says for PerplexityBot today.
3. The page is empty until JavaScript runs
If your text is rendered in the browser, a crawler that reads the HTML response gets the shell: navigation, a footer, and nothing to quote. The page is reachable and still uncitable. The check is whether your title and main text are in the HTML the server sends, before any script runs.
4. A noindex or a wrong canonical
A noindex left over from staging, or a canonical tag pointing at another URL, tells every search system the page is not the one to show. They can come from a meta tag or from an X-Robots-Tag header, and those are set in different places, so finding one does not rule out the other.
5. Nothing is blocking you
If PerplexityBot can fetch your pages, the text is in the HTML, and nothing says noindex, then you are eligible and another source was the better answer for that question. That is a content problem, not an access one, and no robots.txt edit changes it. Rule out the first four before spending a month on it.
What about Perplexity-User?
It fetches a page when somebody in Perplexity asks about it, and Perplexity’s own documentation says it generally ignores robots.txt. A rule for Perplexity-User records a preference and little more. It is not the crawler that decides whether you appear in answers, so it is rarely the reason you are missing.
Check what Perplexity sees
The scan requests your home page as PerplexityBot and as an ordinary browser, compares the two, and reads your robots.txt the way the crawler would. It takes about ten seconds and needs no account.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Crawlers in this guide
- PerplexityBot: Search & citations, run by Perplexity
- Perplexity-User: Fetches when a user asks, run by Perplexity
Where to go next
- Is Cloudflare blocking AI crawlers?
A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way. - Why your site is not showing up in ChatGPT
The common failure modes, in order, with a check you can run for each one. - Do the top 1,000 sites block AI crawlers?
What the most visited sites say in robots.txt, what their servers actually do, and how often the two disagree.