anthropic-ai
An older Anthropic token that its current crawler documentation no longer lists. Kept so a robots.txt that names it is read correctly; it is not scored.
Where this comes from
- Retired by the vendor. Source: support.claude.com
- Robots.txt behaviour undocumented.
- Listed, not scored. It cannot keep a site out of today's answers, so the check shows a block without deducting for it.
In the top 1,000 websites
- 14.1% block it in robots.txt (80 of the 569 sites with a robots.txt), as of 2026-10-12.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
Nothing: the vendor no longer sends this name. A rule for it is harmless and is not scored.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: anthropic-ai
Allow: /To keep it out:
User-agent: anthropic-ai
Disallow: /A rule in robots.txt is a request. Whether this crawler honours it is undocumented. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Also from Anthropic
ClaudeBot· Training onlyClaude-User· Fetches when a user asksClaude-SearchBot· AI search indexClaude-Web· Retired token
Does anthropic-ai get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.