How the top 1,000 websites treat AI crawlers

What robots.txt tells AI crawlers

Each site's policy is read the way its own report reads it: from what its robots.txt says to the 40 crawlers that fetch pages for AI search and answers, and the 9 that gather training data.

• Open to AI79.7% (545)AI search: yes · AI training: yes
• AI search, no AI training8.8% (60)AI search: yes · AI training: no
• Selective3.8% (26)Some AI companies in, others kept out
• Closed to AI7.7% (53)AI search: no · AI training: no

Shares of the 684 sites that answered. Open includes 119 sites with no robots.txt at all, which allows everything. Closed includes 13 hosts that close themselves to every crawler, search engines included: mostly API and CDN hostnames rather than a decision about AI.

Does the server do what robots.txt says?

robots.txt is a request, and the server answers first. After the check's own visit, each homepage was asked for again as four AI crawlers, with each vendor's documented user agent, and the answer was set beside what the site's robots.txt says about that crawler.

Blocked in robots.txt: share of the 684 sites that answered. The server columns: share of the sites whose homepage opened for the check's own request (about 525). Served while blocked: share of those that block that crawler. A refusal is a 401, 403, 429 or 503.

What changed since last month

This is the first edition. From the December 2026 edition on, this section names every site that changed its AI policy, started or stopped blocking one of the four crawlers, or whose server started or stopped refusing a crawler its robots.txt allows.

Look up a site

Every site in the index, as it answered on November 1, 2026. A site that is not in the sample can be checked the same way, free.

The most-blocked AI crawlers

Share of the 684 sites that answered whose robots.txt disallows / for the crawler, by name or through a group it falls under. Every crawler the check knows is in summary.json; what each one is, in the crawler reference.

The files and signals written for AI

Shares of the 684 sites that answered.

Edition by edition

Method

• The same sites every month. The top 1,000 hostnames of Tranco list 8PXYV, the research-standard ranking, downloaded on October 11, 2026 and frozen, so a change in the index is a change in the sites, not in the list. Some are infrastructure hostnames with no website behind them; they count as not answering.
• The same day and the same rules. On the 1st of each month from 06:17 UTC, from a GitHub Actions cloud runner, each site once, six at a time, nothing retried. It is the check this site runs for anyone, scoring rules v3: robots.txt, llms.txt, the homepage and its markdown copy, agents.json and the TDM file as CrawlReadinessBot, then the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, keeping only the status. A site's own report says what the index says about it.
• The four policies. Read from robots.txt's verdicts for the 40 search and answer crawlers and the 9 training crawlers. Open to AI: at most a fifth of the search crawlers blocked and training left mostly open. AI search, no AI training: search open, training blocked (or every training crawler the file names blocked). Closed to AI: every major AI company kept out. Selective: anything else, such as one company left open while the rest are blocked.
• Refused and tested. A refusal is a 401, 403, 429 or 503. A site counts as tested when the server's answers can be read against its robots.txt; when it refused the check's own request too, a crawler's refusal could be a wall against every bot, and the site is left out of the server shares. Each share names what it is a share of.
• Limits. One pass, homepages only. The test sends a crawler's user-agent string, which is what firewall rules match; it does not come from the vendors' IP addresses, so a site that verifies crawlers by address could refuse us and admit the real crawler. Firewalls answer by network and by hour, and a 429 can be a rate limit, so the stricter figure leaves 429-only refusals out. The October 2026 study measured the same sites from a different network with a browser as its baseline, so its numbers are not part of this series.

Download the data

Licensed CC BY 4.0. Cite as “Crawl Readiness, AI Access Index, November 2026, www.crawlreadiness.com/ai-access-index/2026-11.”

• results.csv: one row per site in the sample, with its policy, the server's answer to each of the four crawlers and every signal.
• changes.csv: every change since the edition before, site by site.
• summary.json: every number on this page, with what each is a share of.