# Crawl Readiness > Is your site set up for AI the way you intend? A free check of what AI crawlers can read and what your site tells them they may do, the files that state your policy, a whole-site audit, and Monitor to keep it that way. Crawl Readiness checks how a website presents itself to AI systems and reports what it finds, in plain words: which of 60 AI crawlers may read it according to its robots.txt, what access policy the site states (open to AI, AI search without AI training, closed), whether the server enforces that policy both ways, whether pages are readable without JavaScript, and which machine-readable signals are present: JSON-LD, meta and X-Robots-Tag directives, Open Graph, a sitemap, llms.txt, agents.json, Content-Signal, the TDM reservation, a markdown copy for agents. The 0–100 score counts the crawlers that fetch pages for AI search and answers; blocking training-only crawlers (GPTBot, ClaudeBot, Applebot-Extended, CCBot and the like) costs no points, and a site that restricts AI on purpose is reported as restricted by design, not as failing. No signup is needed for the check. Crawl Readiness is the free front door of Machine Readiness (https://machinereadiness.com), the standard for a website that is readable by AI crawlers, usable by AI agents and verifiably permitted for each use. It describes and measures; it does not promise that AI systems will mention or rank a site. Operated by Tundrastone, an Ontario, Canada business. ## What it does - [The free check](https://www.crawlreadiness.com/): no signup. A 0–100 score, per-crawler status, the access-policy verdict (what the site says, whether its signals agree, whether the server enforces it), a ranked fix list, a shareable report card and an embeddable badge. One URL at a time with no signup; up to 10 at once with a free account. - [The files](https://www.crawlreadiness.com/tools/robots-txt): with a free account, the files a site is missing, generated for it: llms.txt, an AI-crawler robots.txt policy with its Content-Signal and TDM lines, JSON-LD schema. - [The Machine Readiness Audit](https://www.crawlreadiness.com/pricing): $99, one site, once. Every page measured as a crawler receives it (up to 50), the access policy verified at the server both ways, the five fix files written for the site's platform with install steps, an Answer Analysis, a report you can share, and 30 days of weekly re-checks. - [Monitor](https://www.crawlreadiness.com/pricing): $39 a month, or $390 a year. Up to 5 sites re-checked weekly or daily, an email the day a verdict changes, one Answer Analysis per site per month, 1,000 API checks. - [Studio](https://www.crawlreadiness.com/pricing): $79 a month, or $790 a year. 15 sites re-checked weekly or daily, white-label reports with your name and logo, bulk checks of 50 sites, one Answer Analysis per site per month. - [Agency](https://www.crawlreadiness.com/pricing): $199 a month, or $1,990 a year. 25 client sites and more at $8 a site, white-label client reports, client workspaces, bulk checks, one Answer Analysis per client per month, 5,000 API checks. - [Answer Analysis](https://www.crawlreadiness.com/dashboard/monitor): a diagnostic, not a promise. It puts the questions a site's customers ask to live AI search (Perplexity and Google AI Overviews), keeps the pages each answer drew on, checks those sites the same way, and lists what they publish that the site does not. - [Permission check](https://machinereadiness.com/permission): for AI builders. One call answers whether a page may be used for search, AI answers, training or agent action, from every machine-readable signal the site publishes and what the server did when a crawler asked, with a signed, dated receipt. - [MCP server](https://www.crawlreadiness.com/mcp): the check, the generators and the permission call inside Claude Desktop, Cursor, Cline and Windsurf. npm package `crawl-readiness-mcp`. - [WordPress plugin](https://wordpress.org/plugins/grubbomatic-crawl-readiness/): the check inside WordPress, with switches that serve llms.txt, the AI-crawler rules, JSON-LD and agents.json. - [Chrome extension](https://chromewebstore.google.com/detail/jcmhfmdgkkjjjfngbkoonaidbonhhhho): check the site in the current tab. ## Tools - [llms.txt Generator](https://www.crawlreadiness.com/tools/llms-txt): builds an llms.txt from a site's pages (free account) - [Schema Wizard](https://www.crawlreadiness.com/tools/schema-wizard): generates JSON-LD structured data (free account) - [robots.txt Generator](https://www.crawlreadiness.com/tools/robots-txt): writes an AI-crawler policy, with its Content-Signal and TDM lines, and merges it into an existing robots.txt (free account) - [Schema Validator](https://www.crawlreadiness.com/tools/schema-validator): validates JSON-LD on a page (no signup) - [robots.txt Validator](https://www.crawlreadiness.com/tools/robots-validator): checks robots.txt syntax and AI-crawler rules (no signup) - [Content Parity Checker](https://www.crawlreadiness.com/tools/parity): compares what browsers and AI crawlers are served (no signup) ## Docs - [Methodology](https://www.crawlreadiness.com/methodology): every check, its exact point weight, what each grade means, and what the score deliberately does not measure - [The Machine Readiness standard](https://machinereadiness.com/standard): readable, access, usable, proven - [AI crawler reference](https://www.crawlreadiness.com/crawlers): 60 user-agent tokens from 30 vendors, what each one is for, and how to allow or block it - [AI crawler access study, October 2026](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026): what the top 1,000 sites say to AI crawlers and what their servers do - [What is generative engine optimization (GEO)?](https://www.crawlreadiness.com/generative-engine-optimization): an explainer - [Is my website AI ready?](https://www.crawlreadiness.com/is-my-website-ai-ready) - [Blog](https://www.crawlreadiness.com/blog): guides on llms.txt, robots.txt for AI crawlers, blocking AI training without leaving AI answers, and how AI reads a site - [AI Access Index](https://www.crawlreadiness.com/ai-access-index): monthly, the same 1,000 top sites: what robots.txt tells AI crawlers, and whether the server does what it says (data under CC BY-NC 4.0) - [Checked sites](https://www.crawlreadiness.com/checked-sites): every site checked here and its latest result, grouped by grade, unranked - [About](https://www.crawlreadiness.com/about): who runs it, data handling, security contact ## API - Check: `GET https://www.crawlreadiness.com/api/check?url=example.com` returns JSON: score, per-crawler results, the access-policy verdict, detected files and signals, ranked fixes, and a shareable report id. - Permission: `GET https://www.crawlreadiness.com/api/permission?url=https://example.com/page&purpose=train` (purposes: search, answer, train, agent) returns the verdict, the signals behind it, what the server did, and a signed receipt. Also served at https://machinereadiness.com. - Anonymous use is rate limited. A free account provides an API key (`x-api-key` header) with 50 checks a month; Monitor ($39/mo) has 1,000, Studio ($79/mo) 2,500 and Agency ($199/mo) 5,000. ## Markdown copies Every public page on this site is available as markdown (about two thirds smaller than the HTML, so it saves on usage): request it with `Accept: text/markdown`, or add `.md` to its address (https://www.crawlreadiness.com/index.md, https://www.crawlreadiness.com/methodology.md, https://www.crawlreadiness.com/blog/robots-txt-for-ai.md). Each page links its copy with ``. ## Optional - [Full documentation for LLMs](https://www.crawlreadiness.com/llms-full.txt) - [Privacy policy](https://www.crawlreadiness.com/privacy) - [Terms of service](https://www.crawlreadiness.com/terms)