AI crawlers · Meta AI · AI search index · Counted in the score

meta-webindexer

Crawls to improve the quality of Meta AI's search results. Meta says allowing it helps Meta cite and link your pages in Meta AI answers.

Where this comes from

  • Documented by the vendor. Source: developers.facebook.com
  • Honours robots.txt.
  • Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.

In the top 1,000 websites

Not measured in the October 2026 study of the top 1,000 websites.

What blocking it costs

Your pages leave the search index meta-webindexer builds, so Meta AI's product can no longer find or cite them from its own index. The score counts it among the crawlers that feed AI search and answers, so a block costs points.

The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):

User-agent: meta-webindexer
Allow: /

To keep it out:

User-agent: meta-webindexer
Disallow: /

A rule in robots.txt is a request. Meta AI says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

Also from Meta AI

Does meta-webindexer get through on your site?

Run the free check: it reads your robots.txt against all 60 crawlers and tells you which ones can read you, and what to change.