Research

What we measure across the web

Studies run with the same check this site runs for anyone, published with the data, the method and a script to reproduce them.

October 2026·Study
AI crawler access across the top 1,000 websites
One pass over the Tranco top 1,000: which AI crawlers robots.txt blocks, how many servers turn AI crawlers away even when robots.txt allows them, and how far llms.txt, markdown copies and structured data have spread.
AI crawlersrobots.txtdata under CC BY 4.0