Google-NotebookLM
Fetches the individual URLs a notebook's users add as sources. Google now lists it as a former agent, supported until August 2026; the current one is Google-GeminiNotebook. Google says user-triggered fetchers generally ignore robots.txt.
Where this comes from
- Documented by the vendor. Source: developers.google.com Renamed by Google; the newer token is not yet in the check.
- May bypass robots.txt for user-requested fetches.
- Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.
In the top 1,000 websites
- 9.5% block it in robots.txt (54 of the 569 sites with a robots.txt), as of 2026-10-12.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
A rule may not stop it: Google-NotebookLM fetches a page because a person asked about it, and its vendor says such fetches can bypass robots.txt. Where it does honour the rule, the assistant cannot read your page when asked. The score counts it, so a block costs points.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: Google-NotebookLM
Allow: /To keep it out:
User-agent: Google-NotebookLM
Disallow: /A rule in robots.txt is a request. Google AI says a fetch a user asked for may go ahead regardless. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Also from Google AI
Google-Extended· AI answersGoogleOther· Research crawlGemini-Deep-Research· Fetches when a user asksGoogle-CloudVertexBot· Only for the site ownerGoogleOther-Image· Research crawlGoogleOther-Video· Research crawl
Does Google-NotebookLM get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.