Grok’s crawler
Every other crawler on this site has a name you can write in a robots.txt file. Grok does not. xAI publishes no token, no User-Agent string and no IP ranges, which means the rule you were about to copy does nothing — and nothing tells you it did nothing.
| Operator | xAI (Grok) |
|---|---|
| Published robots.txt token | None |
| Published User-Agent | None |
| Published IP ranges | None |
| Crawler documentation | None — xAI has published no crawler policy page |
| Last verified | 2 September 2026 |
What we found
The tokens you will have read about
Search for a way to block Grok and you will be given one of these, usually in a table with twenty other crawlers and no indication that this row is different from the rest:
GrokBotxAI-GrokGrok-DeepSearchxAI-Bot
None of them come from xAI. They are third-party attributions that have propagated between directories, and none has been observed in real traffic. A robots.txt group for a token that nothing sends is not a weak rule or a partial rule; it is an absent one. The file parses, the syntax is valid, the line is simply never matched.
This is worse than having no rule at all, because the rule reads as protection. Anyone auditing the file afterwards — including you, in six months — sees Grok addressed and moves on.
Why no robots.txt rule can reach it
robots.txt is addressed by name. A crawler reads the file, finds the group matching the name it identifies itself with, and follows it; the protocol has no other way to decide which rules apply. That whole mechanism rests on the crawler announcing what it is.
A fetch arriving with an ordinary browser User-Agent is, to robots.txt, an ordinary browser. There is no group for it to match beyond User-agent: *, and a wildcard disallow aimed at Grok would also turn away every crawler you want — Googlebot included. The exclusion protocol is a system of voluntary self-identification, and it has no answer for a client that declines to participate.
What you can actually do
Not much at the robots.txt layer, and it is worth being plain about that rather than offering a rule that would only look like an answer.
Controls that do not depend on self-identification still apply — rate limiting, bot-management rules at your CDN, authentication on anything you do not want fetched. All of them work on request behaviour rather than on a declared name, which is precisely why they still work here. All of them also risk turning away crawlers you want, and that trade-off is the actual decision in front of you. It is not a robots.txt edit.
The one thing we would discourage is treating a directory-listed token as done. If you have a User-agent: GrokBot group in your file today, it is not doing anything, and leaving it there costs you the accurate picture of what your file actually permits.
The operators that do publish an identity
Worth seeing the contrast, because it is what makes xAI’s silence a choice rather than an industry norm. These operators publish both a documented crawler policy and the IP ranges to verify a request really came from them, which is what makes a robots.txt rule for them worth writing:
- GPTBot — OpenAI
- OAI-SearchBot — OpenAI
- ChatGPT-User — OpenAI
- OAI-AdsBot — OpenAI
What your own site says right now
This is the general case of what makes robots.txt hard to reason about: the file states a policy, and whether anything honours it is a different question. We fetch your site as an ordinary browser and again as each AI crawler that does identify itself, then report where the two disagree — the rules you have, versus the access your server actually grants.
We request your robots.txt and your homepage — plus your sitemap, only when robots.txt hides pages from a crawler and we need to say how many. No other page, ever.
Source
- stackfox.co — the third-party field study this entry rests on. xAI publishes nothing to cite, so an observational study is the best available source and is labelled as one.
Re-checked quarterly. xAI may yet publish an official crawler identity, and if it does this page is wrong and will be replaced by a generated one like every other crawler here. Last verified 2 September 2026.