MCP server · Free · Claude, Cursor, Cline & Windsurf

Crawl Readiness MCP Server

Check whether AI crawlers can read a site, then generate the files that fix it — inside Claude, Cursor, Cline and any MCP-compatible client, without leaving the conversation.

What you can ask for

Once it's installed, these are real requests your assistant can carry out — no tab-switching, no copy-pasting URLs into a form.

“Can ChatGPT and Claude actually read acme-inc.com?”

Scores the site against 50+ AI crawlers and hands back a ranked fix list, with the per-crawler robots.txt verdict for each.

“What do AI crawlers see on this page that humans don't?”

Fetches the page as Chrome, GPTBot, ClaudeBot and PerplexityBot in parallel and reports the divergence — including a server that refuses a crawler the robots.txt allows.

“Generate an llms.txt for this site and write it to /public.”

Crawls the sitemap, groups the pages, and produces the finished file — your assistant can save it into the repo it already has open.

“What did my last Answer Analysis find?”

The pages the answers used per provider, the competitors they named and the trend across runs, read straight out of your analyses without opening a dashboard.

“May we train on this page? Can our agent act on that site?”

Reads robots.txt, Content-Signal, Content-Usage, RSL, the TDM reservation, ai.txt and the page's own tags, requests the page as a crawler to see what the server did, and hands back a dated, signed receipt. A reading of published signals, not legal advice.

Nine tools in total, hitting our live API at crawlreadiness.com (the permission check at machinereadiness.com, the same service). The four audit tools and the permission check work anonymously. The three generators and the Answer Analysis read need a free API key — no card, takes a minute.

What is MCP?

Model Context Protocol is Anthropic's open standard for connecting AI assistants to real tools. Any MCP-compatible client — Claude Desktop, Cursor, Cline, Windsurf, Continue — can call an MCP server's tools directly in the conversation. Instead of copy-pasting between your editor and a browser tab, your AI does the work.

Install for Claude Desktop

Open your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add this to the mcpServers block:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel with the four audit tools and the permission check active.

Install for Cursor, Cline, Windsurf, Continue

Same shape — add crawl-readiness to your MCP servers list with npx -y crawl-readiness-mcp as the command. In Cursor: Settings → Features → Model Context Protocol → Add new global MCP server.

Unlock the generator tools

The four audit tools and check_permission work anonymously. The three generators (generate_llms_txt, generate_robots_txt, generate_schema) and get_monitor_trend need an API key — free at crawlreadiness.com/dashboard (50 checks/month, no credit card).

Add the key to the same config:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"],
      "env": {
        "CRAWL_READINESS_API_KEY": "cr_live_xxxxxxxxxxxx"
      }
    }
  }
}

Restart the client. All nine tools are now callable.

The nine tools

check_ai_readinessFree

Score a site 0–100 against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Returns per-crawler robots.txt status, homepage signals, structured-data flags, and a prioritized fix list.

validate_schemaFree

Audit every JSON-LD block on a URL against 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, etc.). Reports required-field errors, recommended-field warnings, and type-specific gotchas.

validate_robotsFree

Line-by-line audit of a robots.txt file. Detects syntax errors, orphan Allow/Disallow lines, unofficial Noindex, wildcard traps, and summarizes AI-bot coverage across 50+ known AI crawlers.

check_content_parityFree

Fetches a page as Chrome, GPTBot, ClaudeBot, and PerplexityBot in parallel. Reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

check_permissionFree

Ask whether a page may be used for search, answers, training or agent action. Reads robots.txt for every crawler that does the job, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and the page's own directives: any reservation wins, and a source that could not be read makes the answer unknown, never allowed. Returns a dated receipt signed with Ed25519, with a hash of every file read. 25 calls a day per network without a key. Documented at machinereadiness.com/permission.

generate_llms_txtAPI key

Crawl a site's sitemap, group pages by section, pull titles and descriptions, and produce a properly formatted llms.txt plus a companion robots.txt snippet — ready to write into /public/.

generate_robots_txtAPI key

AI-crawler-aware robots.txt with four presets: allow-all (public business), search-only, recommended (allow assistants that cite, block training-only bots), and block-all.

generate_schemaAPI key

Detects site name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article JSON-LD — each as a ready-to-paste <script type="application/ld+json"> block.

get_monitor_trendAPI key

Read-only view of your Answer Analysis results: the pages the answers used per provider, the competitors they named, and the trend across runs. Call it with no argument to list your analyses, or pass a name or project id for the detail. Never triggers a run, so it costs nothing.

Example prompts

Once installed, try any of these in your AI client:

“Is my portfolio site AI-readable? URL: yourdomain.com”
“Check if ChatGPT can see acme-inc.com and walk me through the fixes.”
“Validate the JSON-LD on stripe.com — is anything missing?”
“Compare what humans see vs what AI crawlers see on producthunt.com.”
“May I train on https://example.com/research? Check what the site publishes.”
“Generate an llms.txt for my site and save it to /public/llms.txt.”

Your AI can chain tools automatically — audit a site, identify issues, generate the fix files, and write them straight into your project.

Environment variables

Variable	Purpose	Default
`CRAWL_READINESS_API_KEY`	API key for generator tools	none
`CRAWL_READINESS_API_BASE`	Override API base URL (for testing)	https://www.crawlreadiness.com

Source & support

Source code: github.com/Grubbomatic/crawl-readiness-mcp. Issues: github.com/Grubbomatic/crawl-readiness-mcp/issues. Email: support@crawlreadiness.com.

MIT license. The server calls live endpoints at crawlreadiness.com — same rate limits and fair-use rules as the web dashboard.