Grok’s crawler: what xAI does not publish
Most crawlers have a name you can put in robots.txt. Grok does not. xAI publishes no crawler token, User-Agent string or IP ranges, so the rules you may find online do not actually target it.
| Operator | xAI (Grok) |
|---|---|
| Published robots.txt token | None |
| Published User-Agent | None |
| Published IP ranges | None |
| Crawler documentation | None. xAI has published no crawler policy page |
| Last verified | 2 September 2026 |
What we found
The tokens you will have read about
Search for a way to block Grok and you will be given one of these, usually in a table with twenty other crawlers and no indication that this row is different from the rest:
GrokBotxAI-GrokGrok-DeepSearchxAI-Bot
None is published by xAI, and none has been observed in real traffic. A valid robots.txt rule for a token nobody uses is still just an unmatched line.
That can be misleading because the file looks like Grok is covered when it is not.
Can you block Grok in robots.txt?
robots.txt is addressed by name. A crawler reads the file, finds the group matching the name it identifies itself with, and follows it; the protocol has no other way to decide which rules apply. That whole mechanism rests on the crawler announcing what it is.
A fetch arriving with an ordinary browser User-Agent is, to robots.txt, an ordinary browser. There is no group for it to match beyond User-agent: *, and a wildcard disallow aimed at Grok would also turn away every crawler you want, Googlebot included. The exclusion protocol is a system of voluntary self-identification, and it has no answer for a client that declines to participate.
How can you keep Grok out, then?
There is no useful robots.txt control for Grok, so it is better to say that plainly than offer a rule that only looks effective.
Controls that do not depend on self-identification still apply: rate limiting, bot-management rules at your CDN, authentication on anything you do not want fetched. All of them work on request behaviour rather than on a declared name, which is precisely why they still work here. All of them also risk turning away crawlers you want, and that trade-off is the actual decision in front of you. It is not a robots.txt edit.
The one thing we would discourage is treating a directory-listed token as done. If you have a User-agent: GrokBot group in your file today, it is not doing anything, and leaving it there costs you the accurate picture of what your file actually permits.
The operators that do publish an identity
Worth seeing the contrast, because it is what makes xAI’s silence a choice rather than an industry norm. These operators publish both a documented crawler policy and the IP ranges to verify a request really came from them, which is what makes a robots.txt rule for them worth writing:
- GPTBot: OpenAI
- OAI-SearchBot: OpenAI
- ChatGPT-User: OpenAI
- OAI-AdsBot: OpenAI
What your own site says right now
This is the general case of what makes robots.txt hard to reason about: the file states a policy, and whether anything honours it is a different question. We fetch your site as an ordinary browser and again as each AI crawler that does identify itself, then report where the two disagree, comparing the rules you have against the access your server actually grants.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Source
- stackfox.co is the third-party field study this entry rests on. xAI publishes nothing to cite, so an observational study is the best available source and is labelled as one.
Re-checked quarterly. xAI may yet publish an official crawler identity, and if it does this page is wrong and will be replaced by a generated one like every other crawler here. Last verified 2 September 2026.