Research

# What we measure across the web

What we measure across the web, published with the data, the method and a script to reproduce it: how sites let AI crawlers in, turn them away and describe themselves to them.

Studies run with the same check this site runs for anyone, published with the data, the method and a script to reproduce them.

- [October 2026 · Study ## AI crawler access across the top 1,000 websites One pass over the Tranco top 1,000: which AI crawlers robots.txt blocks, how many servers turn AI crawlers away even when robots.txt allows them, and how far llms.txt, markdown copies and structured data have spread. AI crawlers robots.txt data under CC BY 4.0](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026)

---

Source: https://www.crawlreadiness.com/research