Bytespider
ByteDance's crawler, widely reported to gather training data for its models. ByteDance publishes no documentation or IP ranges, and several operators report it fetching pages their robots.txt disallows.
Where this comes from
- Community-listed; no vendor documentation. No vendor page names this token.
- Reported to ignore robots.txt.
- Listed, not scored. It cannot keep a site out of today's answers, so the check shows a block without deducting for it.
In the top 1,000 websites
- 21.1% block it in robots.txt (120 of the 569 sites with a robots.txt), as of 2026-10-12.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
Nothing in today's answers, and nothing in the readiness score: Bytespider only gathers training data, so blocking it is a policy choice about future models, not about being found. The check lists the block without deducting for it.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: Bytespider
Allow: /To keep it out:
User-agent: Bytespider
Disallow: /A rule in robots.txt is a request. Operators report this crawler ignoring it; a server or CDN rule is the only thing that stops it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Does Bytespider get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.