AI crawlers · Google AI · AI answers · Counted in the score

Google-Extended

Not a crawler but a product token: it controls whether content Google already crawls may train future Gemini models and ground answers in the Gemini apps and Vertex AI. Google says it does not affect a site's inclusion or ranking in Google Search; it has no separate user agent.

Where this comes from

  • Documented by the vendor. Source: developers.google.com
  • Honours robots.txt.
  • Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.

In the top 1,000 websites

What blocking it costs

Google AI says this token controls more than training, so a block can keep your pages out of its answers or product features. The score counts it, so a block costs points.

The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):

User-agent: Google-Extended
Allow: /

To keep it out:

User-agent: Google-Extended
Disallow: /

A rule in robots.txt is a request. Google AI says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

Also from Google AI

Does Google-Extended get through on your site?

Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.