GrokBot
The name community lists give xAI's crawler. xAI publishes no crawler documentation, user agent or IP ranges, and one 2026 analysis found Grok traffic arriving without any of these names.
Where this comes from
- Community-listed; no vendor documentation. No vendor page names this token.
- Robots.txt behaviour undocumented.
- Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.
In the top 1,000 websites
- 9.3% block it in robots.txt (53 of the 569 sites with a robots.txt), as of 2026-10-12.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
xAI / Grok says this token controls more than training, so a block can keep your pages out of its answers or product features. The score counts it, so a block costs points.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: GrokBot
Allow: /To keep it out:
User-agent: GrokBot
Disallow: /A rule in robots.txt is a request. Whether this crawler honours it is undocumented. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Also from xAI / Grok
Grok-DeepSearch· Fetches when a user asksxAI-Grok· AI answers
Does GrokBot get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.