Updated 30 September 2026: the Google-Extended section, the training-only list and the recommended file now match how the readiness check has scored since 29 September. The reasoning behind the split is in How to block AI training without disappearing from AI answers.
The core insight most people miss
Every major AI company runs multiple bots, and they do different jobs. Blocking one doesn't block them all. Allowing one doesn't allow them all. If you set up AI-crawler rules without knowing which bot does what, you'll either accidentally block yourself from valuable citations, or accidentally let training crawlers scrape you when you didn't want them to.
Here's the map for 2026.
OpenAI (ChatGPT + related)
- GPTBot — the AI training crawler, and nothing else. Reads your site to potentially include content in future GPT training data. Blocking this doesn't affect ChatGPT answering questions about you today; it affects future model updates.
- OAI-SearchBot — indexes your site for the ChatGPT Search feature. Blocking this hides you from ChatGPT's search results.
- ChatGPT-User — fetches pages in real time when a user asks ChatGPT about a specific URL. OpenAI now says robots.txt rules may not apply to it, because a person asked for the page.
Allow OAI-SearchBot and ChatGPT-User unless you want to be out of ChatGPT altogether. GPTBot is a separate decision, about training only: OpenAI's own documentation says each setting is independent of the others, so you can block GPTBot and keep the other two.
Anthropic (Claude)
- ClaudeBot — AI training crawler, and nothing else.
- Claude-User — real-time browsing when a Claude user asks about a URL.
- Claude-SearchBot — search index crawler.
Same structure as OpenAI, same split: Claude-SearchBot and Claude-User are the answer side, ClaudeBot is training only. Allow the first two; decide the third on its own.
Google AI
Google's AI stack is more complicated because Google has both search and AI products.
- Googlebot — regular search index. If you block this, you disappear from Google Search entirely. Almost nobody wants this. AI Overviews and AI Mode are built from the same index, so this one crawler decides those too.
- Google-Extended — not a crawler but a token that decides what Google may do with pages Googlebot already fetched: train Gemini, and ground answers in the Gemini apps and on Vertex AI. It has no effect on Google Search, AI Overviews or AI Mode. Block it and you leave Gemini's training and its grounding at the same time.
- GoogleOther (and the -Image and -Video variants) — Google's research crawls. Not behind any product you can be found in.
- Gemini-Deep-Research and Google-NotebookLM — fetch pages when a Gemini or NotebookLM user asks.
Recommendation: allow Googlebot (obviously) and the two fetchers. Google-Extended is a genuine trade, not a free opt-out: out of Gemini's training means out of Gemini's grounding too. Our readiness check counts it as an answer crawler for that reason.
Perplexity
- PerplexityBot — index crawler.
- Perplexity-User — live browsing.
Perplexity always cites sources in its answers, so being in its index directly drives referral traffic. Allow both.
The 40+ others
Beyond the big four, there are dozens of smaller AI crawlers: Grok (xAI), Copilot (Microsoft), Meta AI (meta-externalagent), DeepSeek, Mistral, Cohere, CCBot (Common Crawl), DuckAssistBot, Bytespider (ByteDance/TikTok), Diffbot, AI2Bot (Allen AI), Kagibot, PhindBot, YouBot, Applebot-Extended (Apple Intelligence), Amazonbot, PetalBot (Huawei), and more.
Trying to hand-maintain per-bot rules for all of these is a losing battle. The bots come and go faster than you can update your robots.txt. Better strategies:
- Allow everything if you're a public-facing business that wants maximum AI discoverability. Wildcard
Allow: /. - Block the training-only crawlers if you want AI systems to answer questions about you but not train on your content. There are nine of them today (GPTBot, ClaudeBot, Applebot-Extended, CCBot, Bytespider, FacebookBot, cohere-training-data-crawler, AI2Bot and AI2Bot-Dolma), and the
Content-Signaldirective says the same thing in one line. The full reasoning is in How to block AI training without disappearing from AI answers. - Block everything if your content is licensed, paywalled, or otherwise sensitive. Wildcard
Disallow: /.
The new Content-Signal directive
Introduced by Cloudflare and now backed by a growing group of AI companies, Content-Signal lets you express a nuanced preference in one line:
User-agent: *
Content-Signal: ai-train=no, search=yes, ai-input=yesThis says: don't use my content to train models, but DO include me in AI search results and DO fetch me as context when a user asks a question. It's the middle path most businesses want. It goes inside a User-agent group, and it is a stated preference: crawlers that have adopted the convention read it, nothing enforces it, so pair it with explicit rules for the training crawlers you mean.
A recommended robots.txt for most sites
The earlier version of this file allowed GPTBot and ClaudeBot while declaring ai-train=no, which asked for two opposite things. This one says no to training and yes to everything else, consistently. A crawler follows the most specific group that names it, so the nine training groups win for those nine and the catch-all covers the rest.
# Training-only crawlers: out.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /
User-agent: AI2Bot
Disallow: /
User-agent: AI2Bot-Dolma
Disallow: /
# Everyone else, including every search and fetch crawler: in.
User-agent: *
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
Sitemap: https://yourdomain.com/sitemap.xmlThis is the balanced default, and what the generator's Recommended balance preset produces. It allows every crawler that feeds AI search and answers (which drive real referral traffic) while declining training (which doesn't). If you are happy to be in the training sets too, delete the nine groups and the Content-Signal line: User-agent: * plus Allow: / is the whole file.
How to check whether your rules actually work
- Paste your URL into our free robots.txt Validator. It parses your file line-by-line and flags syntax errors, orphaned rules, empty user-agent blocks, and unknown directives.
- Run our AI Readiness Checker — it evaluates your robots.txt against 51 AI crawler user-agents and gives you a pass/fail per bot. Since 29 September the score counts only the crawlers that fetch pages for AI search and answers; training-only blocks are listed under Training only and cost no points.
- For deeper analysis, our robots.txt Generator lets you pick a preset (Allow all AI, Recommended balance, Search & answers only, Block all AI crawlers) and generates a compliant merged file. Your rules for other bots are kept; rules for the AI crawlers you set are replaced, so the file never ends up with two conflicting groups for the same bot.
Common mistakes
- Blocking Googlebot by accident. A
User-agent: * Disallow: /block disappears you from Google Search. Very common on staging sites accidentally promoted to production. - Assuming Google-Extended = Googlebot. They're separate. You can allow one and block the other.
- Using Noindex in robots.txt. Google stopped honoring this in 2019. Use a meta robots tag or X-Robots-Tag header instead.
- Blocking at the CDN without reading the list. A “block AI bots” switch usually takes the search and fetch crawlers down with the training ones, and it does it at the edge, where robots.txt never sees it.
- Not updating for new bots. New AI crawlers emerge every quarter. Recheck your file every 6-12 months.
Bottom line
For 90% of businesses, the right robots.txt for AI is: allow every crawler that feeds AI search and answers; decide about training on its own, and if the answer is no, block the nine training-only crawlers and add the Content-Signal line; then use a tool to keep the file up to date as the ecosystem evolves.
Start with our free robots.txt Generator — pick a preset, download the merged file, upload to your site root. Ten minutes end-to-end.