See what ChatGPT, Claude and Perplexity can read on your site, fix what they can't, and keep it that way.
The exact fix for whatever blocks them, and the tools to keep your site readable as the standards change. New to the terms? What GEO means.
No login required. Checks 50+ AI crawlers.
Is your site staying readable, and what do AI answers use instead?
The readiness check answers “can AI read my site?” once. Watched sites re-check it on a schedule and tell you when the verdict changes. The LLM Monitor asks your customers' questions to live AI search and shows which sites the answers drew on, and what those sites have that yours doesn't.
Two kinds of measurement, labelled on every result: live search (Perplexity and Google AI Overviews search the web for each prompt and show their sources) and model knowledge (what ChatGPT and Claude answer from training, via their APIs). It measures; it does not promise a mention. How the Monitor works →
Add the real questions your buyers might type into an AI assistant — 3 on the free tier, 15 on Pro.
Free covers the model-knowledge columns, run whenever you like. Pro adds Perplexity and Google AI Overviews, the ones that show their sources, and runs them for you weekly or daily.
Every run lists the sites the answers drew on, with their readiness score, and what they have that you don't, so the next fix is a known one.
- Watched sites, re-checked weekly or daily, with an alert when the verdict changes
- Live AI search with sources — Perplexity & Google AI Overviews, weekly or daily
- A real free tier — start without a card
What you can do
- Check — free, no signup. AI-readiness score (0–100), per-crawler allow/deny status, robots.txt / llms.txt / structured-data analysis, a ranked fix list, and a free embeddable badge.
- Fix — free account. Auto-generate the files you're missing — llms.txt, an AI-crawler robots.txt policy, and JSON-LD schema. The ready-to-ship file, not just a to-do list. (Higher volume on Pro.)
- Keep — Pro. Watched sites are re-checked weekly or daily and you are told when the verdict changes, so a CMS update or a new CDN rule can't quietly close the door.
- Prove — free to start. The LLM Monitor asks your customers' questions to AI search and shows which sites the answers drew on, with what they have that you don't. Free covers 1 brand, 3 prompts and 4 runs a month on the model-knowledge columns.(Live search with sources, automatic weekly or daily runs, white-label client reports, bulk checks and client workspaces on Pro/Agency.)
- Run it from your IDE. The Crawl Readiness MCP server puts the check, the generators and the Monitor inside Claude Desktop, Cursor, Cline & Windsurf.
Why AI readiness matters
AI assistants fetch pages to answer questions. If GPTBot, ClaudeBot or PerplexityBot are blocked, or your pages only render with JavaScript, those systems get nothing from your site, whatever else you do. Readiness is the part you control: whether machines can fetch and understand your pages, and whether that matches what you intend. It is not a promise of being mentioned; that depends on far more than readability, which is why the Monitor measures it instead of claiming it.
What Crawl Readiness checks
- 50+ AI crawlers — robots.txt access per bot across OpenAI, Anthropic, Google, Perplexity, xAI, Microsoft, Meta, Mistral, and every other major provider
- llms.txt and llms-full.txt — the emerging AI-native content standard
- Structured data — JSON-LD schema so AI can extract clean facts
- Meta robots and X-Robots-Tag — page-level and header-level indexing directives
- Open Graph tags — supporting signals AI systems use to cite your content
- Sitemap directives and RSS/Atom feeds — content discovery signals
- agents.json — an emerging standard for describing AI-friendly endpoints
How Crawl Readiness works
- Paste any URL into the box above (no signup required)
- We fetch your robots.txt, llms.txt, agents.json and homepage, then evaluate the robots rules for each of 50+ AI crawler user-agents
- You get a 0-100 AI readiness score with per-crawler status and specific fixes ranked by impact
Frequently asked questions
What is AI readiness?
AI readiness measures how easily AI systems like ChatGPT, Claude, and Perplexity can access, read, and understand a website. It looks at robots.txt, llms.txt, structured data, meta tags, and other signals that let AI crawlers index your content.
Which AI crawlers does Crawl Readiness check?
50+ AI crawlers across every major provider — including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, GrokBot, Copilot, Meta-ExternalAgent, Bytespider, Applebot, DeepSeekBot, MistralAI-User, CCBot, DuckAssistBot, and more.
Is Crawl Readiness free?
Yes — you can check any website with no signup, subject to fair-use rate limits. Free accounts get 50 checks per month with detailed reports. Paid Pro ($29/mo) and Agency ($99/mo) plans cover higher volumes.
How do I make my website AI-ready?
Allow the AI crawlers that fetch pages for search and answers in robots.txt (blocking training-only crawlers such as GPTBot is fine), use JSON-LD structured data, set a meta description and Open Graph tags, and make sure key content is server-rendered so crawlers that don't execute JavaScript can still read it. An llms.txt file is optional. Run a check above to see exactly what to fix, or follow our step-by-step guide: how to check if ChatGPT and other AI crawlers can access your website.
What is llms.txt?
llms.txt is an emerging web standard — a Markdown file at the root of your site that summarizes what your site is about in a format AI systems can efficiently read. It complements robots.txt by describing your content rather than restricting it.