How the top 1,000 websites treat AI crawlers
On the 1st of every month, the same 1,000 sites, checked the same way: what each site's robots.txt tells AI crawlers, and what its server actually does when they arrive. It is not a ranking. Open or closed, a policy is a choice; the index measures which choices sites make, and whether their servers keep to them.
The first edition: November 1, 2026
The first edition is measured on November 1, 2026 and published here when the run is done. Every edition answers the same questions about the same 1,000 sites:
- What robots.txt tells AI crawlers: open, search but not training, selective, or closed.
- Whether the server does what robots.txt says: how many refuse a crawler their robots.txt allows, and how many serve one it blocks.
- What changed since the month before, site by site.
- How far llms.txt, Content-Signal, TDM reservations and markdown copies for agents have spread.
It grows out of the October 2026 study, a one-off pass over the same sites.
Method
- The same sites every month. The top 1,000 hostnames of Tranco list 8PXYV, the research-standard ranking, downloaded on October 11, 2026 and frozen, so a change in the index is a change in the sites, not in the list. Some are infrastructure hostnames with no website behind them; they count as not answering.
- The same day and the same rules. On the 1st of each month from 06:17 UTC, from a GitHub Actions cloud runner, each site once, six at a time, nothing retried. It is the check this site runs for anyone: robots.txt, llms.txt, the homepage and its markdown copy, agents.json and the TDM file as
CrawlReadinessBot, then the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, keeping only the status. A site's own report says what the index says about it. - The four policies. Read from robots.txt's verdicts for the 40 search and answer crawlers and the 9 training crawlers. Open to AI: at most a fifth of the search crawlers blocked and training left mostly open. AI search, no AI training: search open, training blocked (or every training crawler the file names blocked). Closed to AI: every major AI company kept out. Selective: anything else, such as one company left open while the rest are blocked.
- Refused and tested. A refusal is a 401, 403, 429 or 503. A site counts as tested when the server's answers can be read against its robots.txt; when it refused the check's own request too, a crawler's refusal could be a wall against every bot, and the site is left out of the server shares. Each share names what it is a share of.
- Limits. One pass, homepages only. The test sends a crawler's user-agent string, which is what firewall rules match; it does not come from the vendors' IP addresses, so a site that verifies crawlers by address could refuse us and admit the real crawler. Firewalls answer by network and by hour, and a 429 can be a rate limit, so the stricter figure leaves 429-only refusals out. The October 2026 study measured the same sites from a different network with a browser as its baseline, so its numbers are not part of this series.
Where does your site stand?
The same check, free, with the fixes: run it on your site. Sites people have checked with the free tool are listed, unranked, in Checked sites.