Bytespider
ByteDance's crawler. A robots.txt rule here is a statement of intent rather than a control, and we will not pretend otherwise.
| Operator | ByteDance |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Widely reported not to |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 13 August 2026 |
What blocking it costs you
What to know
ByteDance publishes no official crawler documentation and no IP ranges, and Bytespider is widely reported to ignore robots.txt.
We still parse and report rules for it, because your declared policy matters and because the absence of a rule is worth seeing. But we label the result honestly: best-effort. If you need this one actually stopped, it has to be stopped at your CDN or firewall, where compliance is not optional.
The robots.txt rules
To block Bytespider:
User-agent: Bytespider
Disallow: /To allow it explicitly — worth doing when your file also contains a blanket User-agent: * disallow, since the most specific matching group wins and a token with no group of its own falls back to the wildcard:
User-agent: Bytespider
Allow: /If neither rule is present, Bytespider is allowed. That is the default, and it is why an accidental block is nearly always something that was added rather than something that was forgotten — usually a snippet copied from a blog post that listed more tokens than the author intended to block.
Is it blocked on your site?
Your robots.txt is only the half you can read. A CDN or bot-management rule can turn Bytespider away while your file still says it is welcome, and that mismatch is invisible from the front of the site. We check both layers separately and tell you which one is doing what.
We request two things: your robots.txt and your homepage. No other pages, ever.