Reference · 51 crawlers · updated 2026-10-12

# AI crawlers: what each one does, and whether to block it

The 51 AI crawler tokens the readiness check knows, grouped by what they do: search indexes, user-requested fetches, answer feeds and training-only crawlers. Each with the vendor's own description, its robots.txt lines, and how many of the top 1,000 sites block it.

Every AI crawler token the readiness check tests, grouped by its job. 35 of the 51 feed AI search or answers and count in the score; the rest gather training data, do research, or are names a vendor no longer uses. Each page gives the vendor's own description where one exists, the exact robots.txt lines, and how many of the top 1,000 websites block that crawler, from [the October 2026 study of the top 1,000 websites](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026).

## How to read this

- **Counted in the score** means blocking the crawler can keep a site out of an AI product's search results or answers, so the check deducts for it.
- **Not scored** means the block is listed but costs nothing: the crawler only trains models, only does research, or no longer runs.
- **Blocked by** is the share of the 569 top-1,000 sites with a robots.txt whose file disallows `/` for that crawler, by name or through a wildcard group.
- **Community-listed; no vendor documentation** marks tokens that come from community lists; the vendor publishes nothing, so treat them as names people use, not promises.

## AI search index

Build the index an AI search product answers from. Block one and you leave that product's results.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`OAI-SearchBot`](https://www.crawlreadiness.com/crawlers/oai-searchbot) | [OpenAI](https://www.crawlreadiness.com/crawlers/vendor/openai) | Counted | 10.2% | Vendor |
| [`Claude-SearchBot`](https://www.crawlreadiness.com/crawlers/claude-searchbot) | [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic) | Counted | 11.1% | Vendor |
| [`PerplexityBot`](https://www.crawlreadiness.com/crawlers/perplexitybot) | [Perplexity](https://www.crawlreadiness.com/crawlers/vendor/perplexity) | Counted | 14.8% | Vendor |
| [`PetalBot`](https://www.crawlreadiness.com/crawlers/petalbot) | Huawei | Counted | 15.6% | Vendor |
| [`DuckAssistBot`](https://www.crawlreadiness.com/crawlers/duckassistbot) | DuckDuckGo AI | Counted | 11.6% | Vendor |
| [`PhindBot`](https://www.crawlreadiness.com/crawlers/phindbot) | Phind | Counted | 10% | Community list |
| [`YouBot`](https://www.crawlreadiness.com/crawlers/youbot) | You.com | Counted | 14.4% | Vendor |
| [`Kagibot`](https://www.crawlreadiness.com/crawlers/kagibot) | Kagi | Counted | 6.9% | Vendor |
| [`AndiBot`](https://www.crawlreadiness.com/crawlers/andibot) | Andi | Counted | 9.7% | Community list |
| [`Timpibot`](https://www.crawlreadiness.com/crawlers/timpibot) | Timpi | Counted | 14.1% | Community list |
| [`GensparkBot`](https://www.crawlreadiness.com/crawlers/gensparkbot) | Genspark | Counted | 6.9% | Community list |

## Fetches when a user asks

Fetch a page when a person asks the assistant about it. Several vendors say these fetches can ignore robots.txt because a user requested them.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`ChatGPT-User`](https://www.crawlreadiness.com/crawlers/chatgpt-user) | [OpenAI](https://www.crawlreadiness.com/crawlers/vendor/openai) | Counted | 12.1% | Vendor |
| [`Claude-User`](https://www.crawlreadiness.com/crawlers/claude-user) | [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic) | Counted | 12% | Vendor |
| [`Gemini-Deep-Research`](https://www.crawlreadiness.com/crawlers/gemini-deep-research) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Counted | 9.8% | Community list |
| [`Google-NotebookLM`](https://www.crawlreadiness.com/crawlers/google-notebooklm) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Counted | 9.5% | Vendor |
| [`Perplexity-User`](https://www.crawlreadiness.com/crawlers/perplexity-user) | [Perplexity](https://www.crawlreadiness.com/crawlers/vendor/perplexity) | Counted | 12.7% | Vendor |
| [`Grok-DeepSearch`](https://www.crawlreadiness.com/crawlers/grok-deepsearch) | [xAI / Grok](https://www.crawlreadiness.com/crawlers/vendor/xai-grok) | Counted | 9.1% | Community list |
| [`meta-externalfetcher`](https://www.crawlreadiness.com/crawlers/meta-externalfetcher) | [Meta AI](https://www.crawlreadiness.com/crawlers/vendor/meta-ai) | Counted | 11.2% | Vendor |
| [`MistralAI-User`](https://www.crawlreadiness.com/crawlers/mistralai-user) | Mistral | Counted | 11.8% | Vendor |
| [`SiderAI`](https://www.crawlreadiness.com/crawlers/siderai) | Sider | Counted | 6.7% | Community list |
| [`Elicit-Bot`](https://www.crawlreadiness.com/crawlers/elicit-bot) | Elicit | Counted | 6.7% | Community list |

## AI answers

Feed answers or product features in ways the vendor describes as more than training, so blocking can cost mentions.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`Google-Extended`](https://www.crawlreadiness.com/crawlers/google-extended) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Counted | 16% | Vendor |
| [`GrokBot`](https://www.crawlreadiness.com/crawlers/grokbot) | [xAI / Grok](https://www.crawlreadiness.com/crawlers/vendor/xai-grok) | Counted | 9.3% | Community list |
| [`xAI-Grok`](https://www.crawlreadiness.com/crawlers/xai-grok) | [xAI / Grok](https://www.crawlreadiness.com/crawlers/vendor/xai-grok) | Counted | 9.3% | Community list |
| [`Copilot`](https://www.crawlreadiness.com/crawlers/copilot) | [Microsoft Copilot](https://www.crawlreadiness.com/crawlers/vendor/microsoft-copilot) | Counted | 9.1% | Community list |
| [`CopilotNative`](https://www.crawlreadiness.com/crawlers/copilotnative) | [Microsoft Copilot](https://www.crawlreadiness.com/crawlers/vendor/microsoft-copilot) | Counted | 9.1% | Community list |
| [`meta-externalagent`](https://www.crawlreadiness.com/crawlers/meta-externalagent) | [Meta AI](https://www.crawlreadiness.com/crawlers/vendor/meta-ai) | Counted | 16.9% | Vendor |
| [`Amazonbot`](https://www.crawlreadiness.com/crawlers/amazonbot) | Amazon | Counted | 13.7% | Vendor |
| [`YandexAdditional`](https://www.crawlreadiness.com/crawlers/yandexadditional) | [Yandex AI](https://www.crawlreadiness.com/crawlers/vendor/yandex-ai) | Counted | 8.3% | Vendor |
| [`YandexAdditionalBot`](https://www.crawlreadiness.com/crawlers/yandexadditionalbot) | [Yandex AI](https://www.crawlreadiness.com/crawlers/vendor/yandex-ai) | Counted | 8.4% | Vendor |
| [`DeepSeekBot`](https://www.crawlreadiness.com/crawlers/deepseekbot) | DeepSeek | Counted | 10.9% | Community list |
| [`cohere-ai`](https://www.crawlreadiness.com/crawlers/cohere-ai) | [Cohere](https://www.crawlreadiness.com/crawlers/vendor/cohere) | Counted | 15.8% | Community list |
| [`Diffbot`](https://www.crawlreadiness.com/crawlers/diffbot) | Diffbot | Counted | 17% | Vendor |
| [`HuggingFace-Bot`](https://www.crawlreadiness.com/crawlers/huggingface-bot) | HuggingFace | Counted | 6.7% | Community list |
| [`Quora-Bot`](https://www.crawlreadiness.com/crawlers/quora-bot) | Quora / Poe | Counted | 7.7% | Community list |

## Training only

Collect training data and nothing else. Blocking them keeps you out of future models and costs nothing in today's answers or in the score.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`GPTBot`](https://www.crawlreadiness.com/crawlers/gptbot) | [OpenAI](https://www.crawlreadiness.com/crawlers/vendor/openai) | Not scored | 18.5% | Vendor |
| [`ClaudeBot`](https://www.crawlreadiness.com/crawlers/claudebot) | [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic) | Not scored | 18.3% | Vendor |
| [`FacebookBot`](https://www.crawlreadiness.com/crawlers/facebookbot) | [Meta AI](https://www.crawlreadiness.com/crawlers/vendor/meta-ai) | Not scored | 13.4% | Community list |
| [`Applebot-Extended`](https://www.crawlreadiness.com/crawlers/applebot-extended) | Apple Intelligence | Not scored | 15.1% | Vendor |
| [`cohere-training-data-crawler`](https://www.crawlreadiness.com/crawlers/cohere-training-data-crawler) | [Cohere](https://www.crawlreadiness.com/crawlers/vendor/cohere) | Not scored | 12.5% | Community list |
| [`CCBot`](https://www.crawlreadiness.com/crawlers/ccbot) | Common Crawl | Not scored | 20.9% | Vendor |
| [`Bytespider`](https://www.crawlreadiness.com/crawlers/bytespider) | Bytedance | Not scored | 21.1% | Community list |
| [`AI2Bot`](https://www.crawlreadiness.com/crawlers/ai2bot) | [Allen AI](https://www.crawlreadiness.com/crawlers/vendor/allen-ai) | Not scored | 13.4% | Vendor |
| [`AI2Bot-Dolma`](https://www.crawlreadiness.com/crawlers/ai2bot-dolma) | [Allen AI](https://www.crawlreadiness.com/crawlers/vendor/allen-ai) | Not scored | 12.1% | Community list |

## Research crawl

Google's research crawls. Their rules affect no product.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`GoogleOther`](https://www.crawlreadiness.com/crawlers/googleother) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Not scored | 11.1% | Vendor |
| [`GoogleOther-Image`](https://www.crawlreadiness.com/crawlers/googleother-image) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Not scored | 7.7% | Vendor |
| [`GoogleOther-Video`](https://www.crawlreadiness.com/crawlers/googleother-video) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Not scored | 7.7% | Vendor |

## Only for the site owner

Crawl only when the site owner asks for it.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`Google-CloudVertexBot`](https://www.crawlreadiness.com/crawlers/google-cloudvertexbot) | [Google AI](https://www.crawlreadiness.com/crawlers/vendor/google-ai) | Not scored | 12.1% | Vendor |

## Retired token

Names the vendor no longer uses, kept so an old robots.txt is read correctly.

| Crawler | Vendor | Score | Blocked by | Documentation |
| --- | --- | --- | --- | --- |
| [`Claude-Web`](https://www.crawlreadiness.com/crawlers/claude-web) | [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic) | Not scored | 14.4% | Retired |
| [`anthropic-ai`](https://www.crawlreadiness.com/crawlers/anthropic-ai) | [Anthropic](https://www.crawlreadiness.com/crawlers/vendor/anthropic) | Not scored | 14.1% | Retired |
| [`NeevaBot`](https://www.crawlreadiness.com/crawlers/neevabot) | Neeva | Not scored | 7.4% | Retired |

## Which of these can read your site?

[Run the free check](https://www.crawlreadiness.com/) for a per-crawler verdict, or build a robots.txt with the [robots.txt generator](https://www.crawlreadiness.com/tools/robots-txt), whose presets follow the same split.

---

Source: https://www.crawlreadiness.com/crawlers