AI Readiness Report
nytimes.com
Restricted by Design
◆80/100
This site restricts most AI crawlers by choice — 15 of 35 AI search and answer crawlers are blocked in robots.txt.
This site appears to intentionally restrict AI crawler access — a valid strategy many publishers and brands choose (for example, keeping content out of AI training). If that’s deliberate, the site is working as intended: the score below measures how open a site is to AI, so a low number here is the expected result of that choice — not a failure to fix.
20
Allowed
15
Blocked
2
Fixes
Blocked crawlers
- OAI-SearchBotOpenAI
- ChatGPT-UserOpenAI
- Claude-UserAnthropic
- Claude-SearchBotAnthropic
- Google-ExtendedGoogle AI
- PerplexityBotPerplexity
- Perplexity-UserPerplexity
- meta-externalagentMeta AI
Recommended fixes
- AI crawlers appear to be intentionally restricted (15 of 35 AI search and answer crawlers blocked in robots.txt)Impact · low
- Your homepage returned HTTP 403 to our crawler — homepage checks were skippedImpact · info
Optional extra
Your site has no llms.txt file
A Markdown summary of nytimes.com for AI systems. Not scored, since few AI crawlers ask for it yet, but quick to add.
Quick fix available
15 AI crawlers blocked by robots.txt
Generate an AI-crawler-aware robots.txt for nytimes.com — pick a preset, download the file.