Free · works in Claude, Cursor, Cline & Windsurf # Crawl Readiness MCP Server Install the Crawl Readiness MCP server and ask Claude, Cursor or Cline whether AI crawlers can read your site — then generate llms.txt, robots.txt and JSON-LD schema straight into your project. Check whether AI crawlers can read a site, then generate the files that fix it — inside Claude, Cursor, Cline and any MCP-compatible client, without leaving the conversation. ## What you can ask for Once it's installed, these are real requests your assistant can carry out — no tab-switching, no copy-pasting URLs into a form. “Can ChatGPT and Claude actually read acme-inc.com?” Scores the site against 50+ AI crawlers and hands back a ranked fix list, with the per-crawler robots.txt verdict for each. “What do AI crawlers see on this page that humans don't?” Fetches the page as Chrome, GPTBot, ClaudeBot and PerplexityBot in parallel and reports the divergence — including a server that refuses a crawler the robots.txt allows. “Generate an llms.txt for this site and write it to /public.” Crawls the sitemap, groups the pages, and produces the finished file — your assistant can save it into the repo it already has open. “What did my LLM Monitor find this week?” The mention rate per provider, the competitors the answers named and a 12-run trend — read straight out of your Monitor projects without opening a dashboard. Eight tools in total, hitting our live API at [crawlreadiness.com](https://www.crawlreadiness.com/). The four audit tools work anonymously. The three generators and the Monitor read need a [free API key](https://www.crawlreadiness.com/dashboard) — no card, takes a minute. ## What is MCP? **Model Context Protocol** is Anthropic's open standard for connecting AI assistants to real tools. Any MCP-compatible client — Claude Desktop, Cursor, Cline, Windsurf, Continue — can call an MCP server's tools directly in the conversation. Instead of copy-pasting between your editor and a browser tab, your AI does the work. ## Install for Claude Desktop Open your config file: - **macOS:** `~/Library/Application Support/Claude/claude_desktop_config.json` - **Windows:** `%APPDATA%\Claude\claude_desktop_config.json` Add this to the `mcpServers` block: ``` { "mcpServers": { "crawl-readiness": { "command": "npx", "args": ["-y", "crawl-readiness-mcp"] } } } ``` Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel with the four audit tools active. ## Install for Cursor, Cline, Windsurf, Continue Same shape — add `crawl-readiness` to your MCP servers list with `npx -y crawl-readiness-mcp` as the command. In Cursor: **Settings → Features → Model Context Protocol → Add new global MCP server**. ## Unlock the generator tools The four audit tools work anonymously. The three generators (`generate_llms_txt`, `generate_robots_txt`, `generate_schema`) and `get_monitor_trend` need an API key — free at [crawlreadiness.com/dashboard](https://www.crawlreadiness.com/dashboard) (50 checks/month, no credit card). Add the key to the same config: ``` { "mcpServers": { "crawl-readiness": { "command": "npx", "args": ["-y", "crawl-readiness-mcp"], "env": { "CRAWL_READINESS_API_KEY": "cr_live_xxxxxxxxxxxx" } } } } ``` Restart the client. All eight tools are now callable. ## The eight tools `check_ai_readiness` Free Score a site 0–100 against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Returns per-crawler robots.txt status, homepage signals, structured-data flags, and a prioritized fix list. `validate_schema` Free Audit every JSON-LD block on a URL against 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, etc.). Reports required-field errors, recommended-field warnings, and type-specific gotchas. `validate_robots` Free Line-by-line audit of a robots.txt file. Detects syntax errors, orphan Allow/Disallow lines, unofficial Noindex, wildcard traps, and summarizes AI-bot coverage across 50+ known AI crawlers. `check_content_parity` Free Fetches a page as Chrome, GPTBot, ClaudeBot, and PerplexityBot in parallel. Reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking. `generate_llms_txt` API key Crawl a site's sitemap, group pages by section, pull titles and descriptions, and produce a properly formatted llms.txt plus a companion robots.txt snippet — ready to write into /public/. `generate_robots_txt` API key AI-crawler-aware robots.txt with four presets: allow-all (public business), search-only, recommended (allow assistants that cite, block training-only bots), and block-all. `generate_schema` API key Detects site name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article JSON-LD — each as a ready-to-paste