AI Readiness Report
theatlantic.com
Restricted by Design
◆80/100
This site restricts most AI crawlers by choice — 14 of 35 AI search and answer crawlers are blocked in robots.txt.
This site appears to intentionally restrict AI crawler access — a valid strategy many publishers and brands choose (for example, keeping content out of AI training). If that’s deliberate, the site is working as intended: the score below measures how open a site is to AI, so a low number here is the expected result of that choice — not a failure to fix.
21
Allowed
14
Blocked
2
Fixes
Blocked crawlers
- Claude-UserAnthropic
- Claude-SearchBotAnthropic
- Google-ExtendedGoogle AI
- PerplexityBotPerplexity
- Perplexity-UserPerplexity
- meta-externalagentMeta AI
- meta-externalfetcherMeta AI
- AmazonbotAmazon
Recommended fixes
- AI crawlers appear to be intentionally restricted (14 of 35 AI search and answer crawlers blocked in robots.txt)Impact · low
- Your homepage returned HTTP 403 to our crawler — homepage checks were skippedImpact · info
Optional extra
Your site has no llms.txt file
A Markdown summary of theatlantic.com for AI systems. Not scored, since few AI crawlers ask for it yet, but quick to add.
Quick fix available
14 AI crawlers blocked by robots.txt
Generate an AI-crawler-aware robots.txt for theatlantic.com — pick a preset, download the file.