Guides
Practical answers to the questions that come before a scan is useful, from which crawlers to allow to whether llms.txt is worth adding. Every crawler fact in them comes from the same registry the scanner reads, currently 26 tokens across 13 operators, verified 19 September 2026.
New to AI crawlers? Your site can work perfectly for people and Google while an AI crawler is turned away at the door. Start with the five things worth checking →
What are you trying to solve?
I want to be found by AI
I want more control
Something isn't working
All guides
- Which AI crawlers should you allow?
The tokens have four jobs. Which ones you want depends on what you are trying to achieve, and the costly mistake is blocking the citation crawler when you meant to block the training one. - AI SEO setup: the five things worth checking
Five layers you can check from outside, each capable of hiding a site. We go through them in the order they tend to break. - Why your site is not showing up in ChatGPT
The common failure modes, in order, with a check you can run for each one. - GPTBot vs OAI-SearchBot: which to block?
OpenAI's two crawlers do opposite jobs. One costs you nothing to block and the other takes you out of ChatGPT's answers. - ClaudeBot vs Claude-SearchBot: which to block?
Anthropic's crawlers split the same way OpenAI's do, and the legacy tokens in most files do nothing at all. - Is Cloudflare blocking AI crawlers?
A correct robots.txt can still leave a site inaccessible. Here is what to look for when Cloudflare is in the way. - Does Google-Extended block AI Overviews?
A common robots.txt mix-up, and the controls that actually decide what AI Overviews can quote. - llms.txt: what it does and doesn't do
Cheap to add and easy to oversell. What the file does, what it does not, and what the available evidence says.
What these guides can and can’t tell you
Nobody outside Google can tell you exactly how to rank. What we can establish from outside your site is whether crawlers can access and read it. That is a different question, but an important one, and it is often answered no without anyone noticing. These guides stay on that side of the line, and say so when something cannot be known.
When you want the answer for one specific site rather than the general case, the scan takes about ten seconds and asks for nothing. If a word on any of these pages is new to you, the glossary has it.