Is Crawl Readiness free to use?

Yes. You can check any website with no signup, subject to fair-use rate limits. Free accounts get 50 checks per month, up to 10 sites at a time, and full detailed reports. The Monitor and Agency plans add scheduled re-checks, alerts and Answer Analyses; the Machine Readiness Audit is $49 once per site.

Is your site set up for AI the way you intend?

Check a site free

Readable where you want it read, closed where you want it closed, and set up so machines get it right. The check shows what 50+ AI crawlers can read, what your access policy says and whether your server enforces it, and the exact fix for whatever is wrong.

Measured against the public Machine Readiness standard.

See the page the way an AI agent receives it.

Check a site

No login required. Checks 50+ AI crawlers.

Reading the board

Allowedrobots.txt lets this crawler fetch your pages.BlockedYour robots.txt shuts it out, so it gets nothing from your site.GPTBotDimmed rows are training-only, research or retired crawlers. They are not counted in the score.

Example reporttheguardian.comchecked 24 Oct 2026

Restricted by Design

Blocks AI crawlers deliberately: a valid choice, not a low score to fix.

Access policyMixed · 10 of 35 AI search and answer crawlers blocked; OpenAI and Google AI left openEnforced: served the 2 allowed crawlers we tested, refused the 2 blocked

Crawler	Operator	Purpose	Status
GPTBotOpenAI	OpenAI	Training only	Allowed
OAI-SearchBotOpenAI	OpenAI	AI search	Allowed
ClaudeBotAnthropic	Anthropic	Training only	Blocked
Claude-SearchBotAnthropic	Anthropic	AI search	Blocked
PerplexityBotPerplexity	Perplexity	AI search	Blocked
Google-ExtendedGoogle AI	Google AI	AI answers	Allowed
GrokBotxAI / Grok	xAI / Grok	AI answers	Allowed

44 more crawlers, every signal and 3 fixes in the full report.

Open the full report

Why AI readiness matters

AI assistants fetch pages to answer questions. If OAI-SearchBot, Claude-SearchBot or PerplexityBot are blocked, or your pages only render with JavaScript, those systems get nothing from your site, whatever else you do.

User-agent: OAI-SearchBot
Disallow: /

Two lines in robots.txt shut one crawler out of the whole site.

Readiness is the part you control: whether machines can fetch and understand your pages, and whether that matches what you intend. It is not a promise of being mentioned; that depends on far more than readability. An Answer Analysis shows which pages AI answers actually used for your customers' questions, so the gap is measured rather than guessed at.

How Crawl Readiness works

Paste any URL into the box above. No signup required.
We fetch your robots.txt, llms.txt, agents.json and homepage, then evaluate the robots rules for each of 50+ AI crawler user-agents.
You get a 0–100 AI readiness score with per-crawler status and specific fixes ranked by impact.

What Crawl Readiness checks

50+ AI crawlers: robots.txt access per bot across OpenAI, Anthropic, Google, Perplexity, xAI, Microsoft, Meta, Mistral, and every other major provider.
Access policy: What the site says machines may do, whether Content-Signal, the TDM reservation and the page agree, and whether the server enforces it, both ways.
llms.txt and llms-full.txt: The emerging AI-native content standard.
Markdown copies for agents: A page served as markdown is about two thirds smaller than its HTML (62–69% on our own pages), so it saves on usage for every agent that fetches it. Every page here serves one: ask with Accept: text/markdown or add .md to the address.
Structured data: JSON-LD schema so AI can extract clean facts.
Meta robots, X-Robots-Tag, Open Graph: Page-level and header-level directives, and the tags AI systems use to cite a page.
Sitemaps, feeds, agents.json: Content discovery signals and the emerging card for AI-friendly endpoints.

What you can do

Check, fix, audit, monitor. The check and the fixes are free.

Check

Free, no signup. A 0–100 score, per-crawler status, the access policy, a ranked fix list and an embeddable badge.

Fix

Free account. The files you are missing, generated: llms.txt, an AI-crawler robots.txt policy with its TDM and Content-Signal lines, JSON-LD schema.

Audit

$49, one site, once. Every page measured against the standard, the files written for your platform, the policy verified at the server, an Answer Analysis, a report you can share, and 30 days of re-checks.

Monitor

$29 a month. Your sites re-checked weekly or daily, an alert the day the verdict or the access policy changes, and one Answer Analysis per site per month.

Agency

$99 a month. Twenty client sites monitored, white-label reports, client workspaces, bulk checks, and an Answer Analysis per client per month.

Run it from your AI assistant or IDE

The MCP server puts the check, the generators and the permission call inside Claude Desktop, Cursor, Cline and Windsurf.

Check from your toolbar

The Chrome extension checks the site you are on in one click. Free.

Monitor · $29 a month, cancel any time

Is your site still set up the way you intend, a month from now?

The check answers it once. Monitor re-checks each site weekly or daily and tells you the day the verdict or the access policy changes: a robots.txt edit, a CDN rule, a plugin update. Once a month, an Answer Analysis shows which pages AI answers used for your customers' questions and what those pages publish that yours don't. Measured, never promised. How it is measured →

See the plans →

Re-checks

Weekly or dailyper site

Each monitored site is re-checked on its schedule. When the verdict or the policy changes, you are told the same day, with the signal that moved.

No driftThe state you want to see every week.

Answer Analysis

Best [category] for small teams?

[competitor.com]/guide: dated, a comparison table, FAQ schema. Each row becomes a line in your fix list.

Perplexity · Google AI Overviews · once a month, or when you ask

Frequently asked questions

What is AI readiness?

AI readiness measures how easily AI systems like ChatGPT, Claude, and Perplexity can access, read, and understand a website. It looks at robots.txt, llms.txt, structured data, meta tags, and other signals that let AI crawlers read your content.

Which AI crawlers does Crawl Readiness check?

50+ AI crawlers across every major provider, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, PerplexityBot, GrokBot, Copilot, Meta-ExternalAgent, Bytespider, Applebot-Extended, DeepSeekBot, MistralAI-User, CCBot, DuckAssistBot, and more.

Is Crawl Readiness free?

Yes. You can check any website with no signup, subject to fair-use rate limits. Free accounts get 50 checks per month, up to 10 sites at a time, with detailed reports. The Monitor ($29/mo) and Agency ($99/mo) plans add scheduled re-checks, alerts and Answer Analyses; the Machine Readiness Audit is $49 once per site.

How do I make my website AI-ready?

If you want AI systems to read your site: allow the AI crawlers that fetch pages for search and answers in robots.txt (blocking training-only crawlers such as GPTBot is fine), use JSON-LD structured data, set a meta description and Open Graph tags, and make sure key content is server-rendered so crawlers that don't execute JavaScript can still read it. An llms.txt file is optional. If you don't want them to, block them and make your other signals say the same; the check shows whether they agree. Run a check above to see exactly what to fix, or follow our step-by-step guide: how to check if ChatGPT and other AI crawlers can access your website.

What is llms.txt?

llms.txt is an emerging web standard: a Markdown file at the root of your site that summarizes what your site is about in a format AI systems can efficiently read. It complements robots.txt by describing your content rather than restricting it.