Google AI's AI crawlers
Google AI documents 6 of these 7; the others are names from community lists. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
Google-Extended | AI answers | Counted | 16% | not tested |
GoogleOther | Research crawl | Not scored | 11.1% | not tested |
Gemini-Deep-Research | Fetches when a user asks | Counted | 9.8% | not tested |
Google-NotebookLM | Fetches when a user asks | Counted | 9.5% | not tested |
Google-CloudVertexBot | Only for the site owner | Not scored | 12.1% | not tested |
GoogleOther-Image | Research crawl | Not scored | 7.7% | not tested |
GoogleOther-Video | Research crawl | Not scored | 7.7% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
Google-Extended · AI answers
Not a crawler but a product token: it controls whether content Google already crawls may train future Gemini models and ground answers in the Gemini apps and Vertex AI. Google says it does not affect a site's inclusion or ranking in Google Search; it has no separate user agent.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
GoogleOther · Research crawl
Google's generic crawler, used by various product teams for one-off crawls and research. Its rules affect no specific product.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Gemini-Deep-Research · Fetches when a user asks
Named in community lists for Gemini's Deep Research feature. Google's current list of user-triggered fetchers does not document it.
Community-listed; no vendor documentation. Its page has the robots.txt lines and what blocking costs.
Google-NotebookLM · Fetches when a user asks
Fetches the individual URLs a notebook's users add as sources. Google now lists it as a former agent, supported until August 2026; the current one is Google-GeminiNotebook. Google says user-triggered fetchers generally ignore robots.txt.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Google-CloudVertexBot · Only for the site owner
Crawls only when a site owner asks for it, to build Vertex AI agents. Google says it has no effect on Google Search or other products.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
GoogleOther-Image · Research crawl
The GoogleOther crawler tuned for fetching public image URLs. Its rules affect no specific product.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
GoogleOther-Video · Research crawl
The GoogleOther crawler tuned for fetching public video URLs. Its rules affect no specific product.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Which of Google AI's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.