AI crawlers · Google AI · Only for the site owner · Not scored

Google-CloudVertexBot

Crawls only when a site owner asks for it, to build Vertex AI agents. Google says it has no effect on Google Search or other products.

Where this comes from

  • Documented by the vendor. Source: developers.google.com
  • Honours robots.txt.
  • Listed, not scored. It cannot keep a site out of today's answers, so the check shows a block without deducting for it.

In the top 1,000 websites

What blocking it costs

Nothing: it only crawls when the site owner sets it up. The check lists the block without deducting for it.

The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):

User-agent: Google-CloudVertexBot
Allow: /

To keep it out:

User-agent: Google-CloudVertexBot
Disallow: /

A rule in robots.txt is a request. Google AI says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

Also from Google AI

Does Google-CloudVertexBot get through on your site?

Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.