Allen AI's AI crawlers
Allen AI documents 1 of these 2; the others are names from community lists. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
AI2Bot | Training only | Not scored | 13.4% | not tested |
AI2Bot-Dolma | Training only | Not scored | 12.1% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
AI2Bot · Training only
Explores certain domains to find web content used to train the Allen Institute's open language models.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
AI2Bot-Dolma · Training only
The same Allen Institute crawl under the name of its Dolma dataset. The institute's page documents AI2Bot; this variant comes from community lists.
Community-listed; no vendor documentation. Its page has the robots.txt lines and what blocking costs.
Opt out of Allen AI's training, stay in its answers
The lines below block only the crawlers that gather training data. The readiness score does not change.
User-agent: AI2Bot
Disallow: /
User-agent: AI2Bot-Dolma
Disallow: /Which of Allen AI's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.