Bytespider: what it does, and what blocking it means
ByteDance's crawler. A robots.txt rule here is a statement of intent rather than a control, and we will not pretend otherwise.
| Operator | ByteDance |
|---|---|
| What it does | Model training |
| Obeys robots.txt | Widely reported not to |
| Governs | AI model training |
| Has a crawler | Yes |
| Last verified | 2 September 2026 |
| Next review | 2 December 2026. We re-check every entry against the operator’s documentation each quarter. |
What happens if you block Bytespider?
Blocking it affects
- ByteDance model training, if it obeys the rule, which it is widely reported not to
Does Bytespider obey robots.txt?
ByteDance publishes no crawler documentation and no IP ranges, and Bytespider is widely reported to ignore robots.txt.
We still parse and report rules for it, because your declared policy matters and the absence of a rule is worth seeing. But we label the result best-effort. If you need this one actually stopped, stop it at your CDN or firewall, where compliance is not optional.
How do you spot Bytespider in your logs?
To find Bytespider in your logs, match the User-Agent shown on this page. The robots.txt token is what the crawler reads, not what appears in the request.
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)Anyone can send that string, and ByteDance publishes no address ranges for Bytespider, so there is no way to confirm from outside that a request carrying it is really theirs.
Worth knowing about Bytespider
ByteDance publishes no documentation for Bytespider, so the User-Agent above comes from field observations and may not match every request.
How do you block or allow Bytespider in robots.txt?
To block Bytespider:
User-agent: Bytespider
Disallow: /To allow it explicitly, use an Allow: / rule if your file has a blanket User-agent: * disallow:
User-agent: Bytespider
Allow: /With neither rule, Bytespider is allowed by default.
Not sure how Bytespider fits with the others? Build the whole file, or see which AI crawlers to allow.
How many sites block Bytespider?
In our scan of the Tranco top 1,000 (1 to 2 October 2026, 499 sites measured), 17.2% disallowed Bytespider in robots.txt. We read the rules for Bytespider but did not request pages as it, so how many servers would refuse it is not measured.
Is your site blocking Bytespider?
A CDN or bot-management rule can still block Bytespider even when robots.txt says it is welcome. You cannot see that mismatch from the front of the site.
We request your robots.txt and homepage. We read your sitemap only when robots.txt hides pages from a crawler and we need to measure how much is affected. No other page.
Guides about Bytespider
- How to block AI crawlers and stay in AI search
The robots.txt that refuses training and keeps citations, built from the registry, plus what to do about the crawlers that do not read the file.
Which other crawlers do the same job as Bytespider?
- GPTBot: Model training, run by OpenAI
- ClaudeBot: Model training, run by Anthropic
- Google-Extended: Model training, run by Google
- Applebot-Extended: Model training, run by Apple
- meta-externalagent: Model training, run by Meta
- CCBot: Open training corpus, run by Common Crawl
- MistralAI-Training: Model training, run by Mistral