AI Readiness Report
theguardian.com
Restricted by Design
◆67/100
This site restricts 11 of 40 AI search and answer crawlers by choice, in robots.txt.
Access policyMixed · OpenAI and Google AI left open · enforced
This site appears to restrict AI crawlers on purpose — a valid choice many publishers and brands make (for example, keeping content out of AI training, or reserving it for the AI companies they have a deal with). If that’s deliberate, the site is working as intended. Read the verdict, not the number: the score still counts each blocked crawler, so a lower score is the expected result of that choice, not a failure to fix.
29
Allowed
11
Blocked
3
Fixes
Blocked crawlers
- Claude-UserAnthropic
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
- meta-externalagentMeta AI
- AmazonbotAmazon
- Amzn-SearchBotAmazon
- PetalBotHuawei
- YandexAdditionalYandex AI
- +10 morein the full check
Recommended fixes
- AI crawlers appear to be intentionally restricted (11 of 40 AI search and answer crawlers blocked in robots.txt)Impact · low
- No schema.org / JSON-LD structured data foundImpact · medium
- No Open Graph tags foundImpact · low
Optional extra
Your site has no llms.txt file
A Markdown summary of theguardian.com for AI systems. Not scored, since few AI crawlers ask for it yet, but quick to add.
Your choice
Want a different policy?
Generate a robots.txt for theguardian.com that states the policy you choose: open, AI search without training, or closed.
Quick fix available
No JSON-LD structured data detected
Auto-generate Organization + WebSite schema for theguardian.com — paste one snippet into your <head>.