xAI / Grok's AI crawlers
xAI / Grok publishes no crawler documentation; these names come from community lists, so treat them as what people write in robots.txt rather than as the vendor's promise. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
GrokBot | AI answers | Counted | 9.3% | not tested |
Grok-DeepSearch | Fetches when a user asks | Counted | 9.1% | not tested |
xAI-Grok | AI answers | Counted | 9.3% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
GrokBot · AI answers
The name community lists give xAI's crawler. xAI publishes no crawler documentation, user agent or IP ranges, and one 2026 analysis found Grok traffic arriving without any of these names.
Community-listed; no vendor documentation. Its page has the robots.txt lines and what blocking costs.
Grok-DeepSearch · Fetches when a user asks
A community-listed name for Grok's DeepSearch fetches. xAI documents nothing about it.
Community-listed; no vendor documentation. Its page has the robots.txt lines and what blocking costs.
xAI-Grok · AI answers
Another community-listed name for xAI's crawling. xAI documents nothing about it.
Community-listed; no vendor documentation. Its page has the robots.txt lines and what blocking costs.
Which of xAI / Grok's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.