Check whether ChatGPT, Claude, Perplexity, Google AI, Grok, Copilot, and 30+ other AI crawlers can actually access and understand a website — and get the exact fix if they can't.
No login required. Checks 30+ AI crawlers.
Get the Crawl Readiness ExtensionCheck any site's AI readiness in one click — free on the Chrome Web StoreWhy AI readiness matters
As AI chat and answer engines replace traditional search, your website's visibility depends on whether AI systems can actually access and understand your content. If GPTBot, ClaudeBot, or PerplexityBot are blocked or can't parse your pages, users asking those AI tools about your business will get nothing — or worse, your competitor's answer.
What Crawl Readiness checks
- 32 AI crawlers — robots.txt access per bot across OpenAI, Anthropic, Google, Perplexity, xAI, Microsoft, Meta, Mistral, and every other major provider
- llms.txt and llms-full.txt — the emerging AI-native content standard
- Structured data — JSON-LD schema so AI can extract clean facts
- Meta robots and X-Robots-Tag — page-level and header-level indexing directives
- Open Graph tags — supporting signals AI systems use to cite your content
- Sitemap directives and RSS/Atom feeds — content discovery signals
- agents.json — an emerging standard for describing AI-friendly endpoints
How Crawl Readiness works
- Paste any URL into the box above (no signup required)
- We fetch robots.txt, llms.txt, agents.json, and your homepage as each of 32 AI bots
- You get a 0-100 AI readiness score with per-crawler status and specific fixes ranked by impact
Frequently asked questions
What is AI readiness?
AI readiness measures how easily AI systems like ChatGPT, Claude, and Perplexity can access, read, and understand a website. It looks at robots.txt, llms.txt, structured data, meta tags, and other signals that let AI crawlers index your content.
Which AI crawlers does Crawl Readiness check?
32 AI crawlers across every major provider — including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, GrokBot, Copilot, Meta-ExternalAgent, Bytespider, Applebot, DeepSeekBot, MistralAI-User, CCBot, DuckAssistBot, and more.
Is Crawl Readiness free?
Yes — you can check any website with no signup, up to 10 checks per day per IP. Free accounts get 50 checks per month with detailed reports. Paid Pro ($29/mo) and Agency ($99/mo) plans cover higher volumes.
How do I make my website AI-ready?
Allow AI crawlers in robots.txt, add an llms.txt file describing your site, use JSON-LD structured data, set Open Graph tags, and make sure key content is server-rendered so crawlers that don't execute JavaScript can still read it. Run a check above to see exactly what to fix on any site.
What is llms.txt?
llms.txt is an emerging web standard — a Markdown file at the root of your site that summarizes what your site is about in a format AI systems can efficiently read. It complements robots.txt by describing your content rather than restricting it.