AI crawlers · Mistral · Fetches when a user asks · Counted in the score

MistralAI-User

Visits a page when a user's question in Mistral's assistant needs it, and links to the source; Mistral says it is not used for crawling the web automatically or for training.

Where this comes from

In the top 1,000 websites

What blocking it costs

A rule may not stop it: MistralAI-User fetches a page because a person asked about it, and its vendor says such fetches can bypass robots.txt. Where it does honour the rule, the assistant cannot read your page when asked. The score counts it, so a block costs points.

The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):

User-agent: MistralAI-User
Allow: /

To keep it out:

User-agent: MistralAI-User
Disallow: /

A rule in robots.txt is a request. Mistral says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

Also from Mistral

Does MistralAI-User get through on your site?

Run the free check: it reads your robots.txt against all 60 crawlers and tells you which ones can read you, and what to change.