AI crawler access across the top 1,000 websites

The number nobody publishes: allowed on paper, refused at the door

For each of the 549 sites whose homepage opened for a browser, the homepage was asked for again four times, as the four crawlers that matter most to being found or trained on, and only the status was kept. A 401, 403, 429 or 503 counts as refused. Then that answer was set beside what the site's own robots.txt says about the same crawler.

78 sites (14.2%) turn at least one of the four away while their robots.txt allows it. Firewalls and CDN bot rules sit in front of robots.txt and answer first, and nothing in the file tells the site owner it is happening.

One caveat belongs next to that number. A 429 can mean rate limiting rather than a rule against the crawler, and 14 of the 78 sites answered the crawlers with a 429 and nothing else. Leave those out and the figure is 64 sites (11.7%): refused outright, with a 401, 403 or 503. Both numbers are in the table, and in the data.

Shares are of the 549 sites that answered a browser with a 2xx. The last column is the reverse case: robots.txt says no, but the server hands the page over anyway, which is what happens when a crawler ignores the file or a site never enforces it.

What robots.txt says

569 of the 688 sites that answered have a robots.txt (82.7%). Among those:

• 25.1% block at least one crawler that fetches pages for AI search or answers (143 sites); 10.2% block more than half of them.
• 26% block at least one training-only crawler (GPTBot, ClaudeBot, CCBot, Bytespider and the like).
• Only 2.6% (15 sites) follow the policy that keeps a site in the answers while opting out of training: block the training crawlers, leave the search and answer crawlers open.
• 62.4% name a sitemap in robots.txt; 2.5% carry a Content-Signal line.

The most-blocked crawlers

Every one of the 51 tokens the check knows is in crawlers.csv. A crawler is “blocked” when the site's robots.txt disallows / for it, by name or through a wildcard group it falls under.

How ready the top sites are

Scored the way the methodology describes (scoring v2: only crawlers that feed AI search and answers count; training-only blocks and the emerging files do not), the 688 sites that answered average 84.5 out of 100, median 90.

• AI Ready: 355 sites. Open to the crawlers that matter, with the metadata to be understood.
• Partly Checked: 126 sites (21.5% with an unreadable homepage). The homepage answered our check with a 403 or similar: the same door-closing the parity test measures, seen from the other side.
• Restricted by Design: 106 sites (15.4%). They block AI crawlers deliberately; a low score there is a choice, not a failure.
• JavaScript-Dependent: 88 sites (12.8%). Little readable text without running scripts, which most AI crawlers do not.
• Partial Access: 13 sites, blocking some of the search and answer crawlers but not most.

The emerging files, and the old ones

• llms.txt: 15.1% (104 sites) serve a real one, a text file that starts with a Markdown heading. A looser count, any 2xx answer at /llms.txt, gives 34.3%, because a third of the top sites answer every path with their homepage; that gap is why “has an llms.txt” claims should be read carefully. llms-full.txt: 21.9% by the loose count.
• A markdown copy for agents: 3.5% (24 sites) answer Accept: text/markdown with markdown; 10 link one from the page head.
• agents.json: 0.3% (2 sites).
• Forms an agent can use: 254 homepages have a form; 1 of them carry WebMCP's tool attributes.
• JSON-LD structured data: 36.2%. Meta description: 65.4%. Open Graph title: 54.4%. An RSS or Atom feed: 8.9%. A noindex on the homepage: 1.5%.

Method

• Sample. Tranco list 8PXYV (https://tranco-list.eu/list/8PXYV), top 1,000, downloaded 2026-10-11. The Tranco list is the research-standard ranking (tranco-list.eu), so the sample is defined by someone else and can be reproduced.
• What was fetched, once per site. robots.txt, llms.txt, llms-full.txt, /.well-known/agents.json, the homepage, and the homepage again with Accept: text/markdown, all as CrawlReadinessBot/1.0; then the homepage as a Chrome browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, using each vendor's documented user agent, keeping only the status. About ten requests a site, six sites at a time, nothing retried.
• Definitions. “Refused” is a 401, 403, 429 or 503; a 429 can be rate limiting rather than a rule against the crawler, so each number is also given without sites whose only refusals were 429s. “Allowed in robots.txt” means the file does not disallow / for that crawler; a missing robots.txt allows everything. Shares use the denominator that applies: the 688 sites that answered, the 569 with a robots.txt, or the 549 whose homepage opened for a browser.
• Limits. One day, one pass, homepages only. The parity test measures how a server answers a user-agent string, which is what a firewall rule keys on; it does not come from the vendors' IP ranges, so a site that verifies crawlers by IP could refuse us and admit the real crawler. 312 hostnames in the sample never answered, mostly CDN and infrastructure names that are in the top 1,000 by traffic without being websites. The llms.txt count was re-fetched and filtered for soft 404s; the other presence counts are as the check reports them.
• Reproduce it. parity.mjs re-runs the refused-at-the-door test on any list of sites: one file, Node 18 or newer, no dependencies. Save the Tranco list as list.csv and run node parity.mjs list.csv (add --limit 25 for a trial). It prints the table above. Expect small differences: firewalls answer by address and by hour, and rate limits (429) come and go. The readiness score and the presence counts come from the check itself, which the script does not include; every site's values are in results.csv. The same run against next month's list is the plan.

Download the data

Licensed CC BY 4.0. Cite as “Crawl Readiness, AI crawler access across the top 1,000 websites, 2026-10-12.”

• results.csv: one row per site, with the score, the grade, what robots.txt says about each of the four parity crawlers, the five homepage statuses and every presence flag.
• crawlers.csv: one row per crawler token, with blocked counts and, for the four tested, the parity numbers.
• summary.json: every aggregate on this page; llms-verify.json: why each reported llms.txt was or was not counted.
• parity.mjs: the script from the Method section, to re-run the parity test on this list or any other.