[AI crawlers](https://www.crawlreadiness.com/crawlers) · Anthropic · Training only · Not scored

# ClaudeBot

ClaudeBot is Anthropic's training only crawler token. Blocking it costs nothing in the AI readiness score. 18.3% of the top 1,000 websites with a robots.txt block it.

Collects web content that could contribute to training Claude's models. Anthropic says its bots honour robots.txt and support Crawl-delay.

## Where this comes from

- **Documented by the vendor.** Source: [support.claude.com](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- **Honours robots.txt.** Published addresses: [claude.com/crawling/bots.json](https://claude.com/crawling/bots.json).
- **Listed, not scored.** It cannot keep a site out of today's answers, so the check shows a block without deducting for it.

## In the top 1,000 websites

- **18.3% block it in robots.txt** (104 of the 569 sites with a robots.txt), as of 2026-10-12.
- **14.8% refuse it at the server**, answering its user agent with a 401, 403, 429 or 503 while a browser gets the page.
- **10.9% allow it in robots.txt and refuse it anyway** (60 sites): a firewall or CDN rule the site owner may not know about. The reverse also happens: 49 sites block it in robots.txt and still serve it.
- Method, limits and the full data: [the October 2026 study of the top 1,000 websites](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026).

## What blocking it costs

Nothing in today's answers, and nothing in the readiness score: ClaudeBot only gathers training data, so blocking it is a policy choice about future models, not about being found. The check lists the block without deducting for it.

## The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a `User-agent: *` block):

```
User-agent: ClaudeBot
Allow: /
```

To keep it out:

```
User-agent: ClaudeBot
Disallow: /
```

A rule in robots.txt is a request. Anthropic says this crawler honours it. The [robots.txt generator](https://www.crawlreadiness.com/tools/robots-txt) writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

## Also from [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic)

- [`Claude-User`](https://www.crawlreadiness.com/crawlers/claude-user) · Fetches when a user asks
- [`Claude-SearchBot`](https://www.crawlreadiness.com/crawlers/claude-searchbot) · AI search index
- [`Claude-Web`](https://www.crawlreadiness.com/crawlers/claude-web) · Retired token
- [`anthropic-ai`](https://www.crawlreadiness.com/crawlers/anthropic-ai) · Retired token

## Does ClaudeBot get through on your site?

[Run the free check](https://www.crawlreadiness.com/): it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.

---

Source: https://www.crawlreadiness.com/crawlers/claudebot