Anthropic's AI crawlers
Anthropic documents 3 of these 5; the others are names from community lists. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
ClaudeBot | Training only | Not scored | 18.3% | 10.9% |
Claude-User | Fetches when a user asks | Counted | 12% | not tested |
Claude-SearchBot | AI search index | Counted | 11.1% | not tested |
Claude-Web | Retired token | Not scored | 14.4% | not tested |
anthropic-ai | Retired token | Not scored | 14.1% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
ClaudeBot · Training only
Collects web content that could contribute to training Claude's models. Anthropic says its bots honour robots.txt and support Crawl-delay.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Claude-User · Fetches when a user asks
Fetches a site when a person asks Claude a question that needs web retrieval, so Claude can answer from the page and cite it.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Claude-SearchBot · AI search index
Navigates the web to improve the quality of Claude's search results, so answers can point to relevant, current pages.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Claude-Web · Retired token
An older Anthropic token that its current crawler documentation no longer lists. Kept so a robots.txt that names it is read correctly; it is not scored.
Retired by the vendor. Its page has the robots.txt lines and what blocking costs.
anthropic-ai · Retired token
An older Anthropic token that its current crawler documentation no longer lists. Kept so a robots.txt that names it is read correctly; it is not scored.
Retired by the vendor. Its page has the robots.txt lines and what blocking costs.
Opt out of Anthropic's training, stay in its answers
The lines below block only the crawler that gather training data. Claude-User, Claude-SearchBot stay open, so the site is still found and cited. The readiness score does not change.
User-agent: ClaudeBot
Disallow: /Which of Anthropic's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.