Reference · 51 crawlers · updated 2026-10-12

AI crawlers: what each one does, and whether to block it

Every AI crawler token the readiness check tests, grouped by its job. 35 of the 51 feed AI search or answers and count in the score; the rest gather training data, do research, or are names a vendor no longer uses. Each page gives the vendor's own description where one exists, the exact robots.txt lines, and how many of the top 1,000 websites block that crawler, from the October 2026 study of the top 1,000 websites.

How to read this

  • Counted in the score means blocking the crawler can keep a site out of an AI product's search results or answers, so the check deducts for it.
  • Not scored means the block is listed but costs nothing: the crawler only trains models, only does research, or no longer runs.
  • Blocked by is the share of the 569 top-1,000 sites with a robots.txt whose file disallows / for that crawler, by name or through a wildcard group.
  • Community-listed; no vendor documentation marks tokens that come from community lists; the vendor publishes nothing, so treat them as names people use, not promises.

Fetches when a user asks

Fetch a page when a person asks the assistant about it. Several vendors say these fetches can ignore robots.txt because a user requested them.

CrawlerVendorScoreBlocked byDocumentation
ChatGPT-UserOpenAICounted12.1%Vendor
Claude-UserAnthropicCounted12%Vendor
Gemini-Deep-ResearchGoogle AICounted9.8%Community list
Google-NotebookLMGoogle AICounted9.5%Vendor
Perplexity-UserPerplexityCounted12.7%Vendor
Grok-DeepSearchxAI / GrokCounted9.1%Community list
meta-externalfetcherMeta AICounted11.2%Vendor
MistralAI-UserMistralCounted11.8%Vendor
SiderAISiderCounted6.7%Community list
Elicit-BotElicitCounted6.7%Community list

AI answers

Feed answers or product features in ways the vendor describes as more than training, so blocking can cost mentions.

CrawlerVendorScoreBlocked byDocumentation
Google-ExtendedGoogle AICounted16%Vendor
GrokBotxAI / GrokCounted9.3%Community list
xAI-GrokxAI / GrokCounted9.3%Community list
CopilotMicrosoft CopilotCounted9.1%Community list
CopilotNativeMicrosoft CopilotCounted9.1%Community list
meta-externalagentMeta AICounted16.9%Vendor
AmazonbotAmazonCounted13.7%Vendor
YandexAdditionalYandex AICounted8.3%Vendor
YandexAdditionalBotYandex AICounted8.4%Vendor
DeepSeekBotDeepSeekCounted10.9%Community list
cohere-aiCohereCounted15.8%Community list
DiffbotDiffbotCounted17%Vendor
HuggingFace-BotHuggingFaceCounted6.7%Community list
Quora-BotQuora / PoeCounted7.7%Community list

Training only

Collect training data and nothing else. Blocking them keeps you out of future models and costs nothing in today's answers or in the score.

CrawlerVendorScoreBlocked byDocumentation
GPTBotOpenAINot scored18.5%Vendor
ClaudeBotAnthropicNot scored18.3%Vendor
FacebookBotMeta AINot scored13.4%Community list
Applebot-ExtendedApple IntelligenceNot scored15.1%Vendor
cohere-training-data-crawlerCohereNot scored12.5%Community list
CCBotCommon CrawlNot scored20.9%Vendor
BytespiderBytedanceNot scored21.1%Community list
AI2BotAllen AINot scored13.4%Vendor
AI2Bot-DolmaAllen AINot scored12.1%Community list

Research crawl

Google's research crawls. Their rules affect no product.

CrawlerVendorScoreBlocked byDocumentation
GoogleOtherGoogle AINot scored11.1%Vendor
GoogleOther-ImageGoogle AINot scored7.7%Vendor
GoogleOther-VideoGoogle AINot scored7.7%Vendor

Only for the site owner

Crawl only when the site owner asks for it.

CrawlerVendorScoreBlocked byDocumentation
Google-CloudVertexBotGoogle AINot scored12.1%Vendor

Retired token

Names the vendor no longer uses, kept so an old robots.txt is read correctly.

CrawlerVendorScoreBlocked byDocumentation
Claude-WebAnthropicNot scored14.4%Retired
anthropic-aiAnthropicNot scored14.1%Retired
NeevaBotNeevaNot scored7.4%Retired

Which of these can read your site?

Run the free check for a per-crawler verdict, or build a robots.txt with the robots.txt generator, whose presets follow the same split.