AI Readiness Report
theguardian.com
Restricted by Design
◆67/100
This site restricts most AI crawlers by choice — 10 of 35 AI search and answer crawlers are blocked in robots.txt.
This site appears to intentionally restrict AI crawler access — a valid strategy many publishers and brands choose (for example, keeping content out of AI training). If that’s deliberate, the site is working as intended: the score below measures how open a site is to AI, so a low number here is the expected result of that choice — not a failure to fix.
25
Allowed
10
Blocked
3
Fixes
Blocked crawlers
- Claude-UserAnthropic
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
- meta-externalagentMeta AI
- AmazonbotAmazon
- PetalBotHuawei
- YandexAdditionalYandex AI
- YandexAdditionalBotYandex AI
Recommended fixes
- AI crawlers appear to be intentionally restricted (10 of 35 AI search and answer crawlers blocked in robots.txt)Impact · low
- No schema.org / JSON-LD structured data foundImpact · medium
- No Open Graph tags foundImpact · low
Optional extra
Your site has no llms.txt file
A Markdown summary of theguardian.com for AI systems. Not scored, since few AI crawlers ask for it yet, but quick to add.
Quick fix available
10 AI crawlers blocked by robots.txt
Generate an AI-crawler-aware robots.txt for theguardian.com — pick a preset, download the file.
Quick fix available
No JSON-LD structured data detected
Auto-generate Organization + WebSite schema for theguardian.com — paste one snippet into your <head>.