New · v0.1.0

Crawl Readiness MCP Server

AI SEO tools inside Claude, Cursor, Cline, and any MCP-compatible client. Audit a site, then generate the files that fix it — without leaving the conversation.

What is MCP?

Model Context Protocol is Anthropic's open standard for connecting AI assistants to real tools. Any MCP-compatible client — Claude Desktop, Cursor, Cline, Windsurf, Continue — can call an MCP server's tools directly in the conversation. Instead of copy-pasting between your editor and a browser tab, your AI does the work.

The Crawl Readiness MCP server exposes seven tools that hit our live API at crawlreadiness.com: four audit tools (free, no signup) and three generator tools (free API key).

Install for Claude Desktop

Open your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add this to the mcpServers block:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel with the four audit tools active.

Install for Cursor, Cline, Windsurf, Continue

Same shape — add crawl-readiness to your MCP servers list with npx -y crawl-readiness-mcp as the command. In Cursor: Settings → Features → Model Context Protocol → Add new global MCP server.

Unlock the generator tools

The four audit tools work anonymously. The three generator tools (generate_llms_txt, generate_robots_txt, generate_schema) need an API key — free at crawlreadiness.com/dashboard (50 checks/month, no credit card).

Add the key to the same config:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"],
      "env": {
        "CRAWL_READINESS_API_KEY": "cr_live_xxxxxxxxxxxx"
      }
    }
  }
}

Restart the client. All seven tools are now callable.

The seven tools

check_ai_readinessFree

Score a site 0–100 against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Returns per-crawler robots.txt status, homepage signals, structured-data flags, and a prioritized fix list.

validate_schemaFree

Audit every JSON-LD block on a URL against 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, etc.). Reports required-field errors, recommended-field warnings, and type-specific gotchas.

validate_robotsFree

Line-by-line audit of a robots.txt file. Detects syntax errors, orphan Allow/Disallow lines, unofficial Noindex, wildcard traps, and summarizes AI-bot coverage across 50+ known AI crawlers.

check_content_parityFree

Fetches a page as Chrome, GPTBot, ClaudeBot, and PerplexityBot in parallel. Reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

generate_llms_txtAPI key

Crawl a site's sitemap, group pages by section, pull titles and descriptions, and produce a properly formatted llms.txt plus a companion robots.txt snippet — ready to write into /public/.

generate_robots_txtAPI key

AI-crawler-aware robots.txt with four presets: allow-all (public business), search-only, recommended (allow assistants that cite, block training-only bots), and block-all.

generate_schemaAPI key

Detects site name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article JSON-LD — each as a ready-to-paste <script type="application/ld+json"> block.

Example prompts

Once installed, try any of these in your AI client:

“Is my portfolio site AI-readable? URL: yourdomain.com”
“Check if ChatGPT can see acme-inc.com and walk me through the fixes.”
“Validate the JSON-LD on stripe.com — is anything missing?”
“Compare what humans see vs what AI crawlers see on producthunt.com.”
“Generate an llms.txt for my site and save it to /public/llms.txt.”

Your AI can chain tools automatically — audit a site, identify issues, generate the fix files, and write them straight into your project.

Environment variables

Variable	Purpose	Default
`CRAWL_READINESS_API_KEY`	API key for generator tools	none
`CRAWL_READINESS_API_BASE`	Override API base URL (for testing)	https://www.crawlreadiness.com

Source & support

Source code: github.com/Grubbomatic/Crawl-Readiness/tree/main/mcp-server. Issues: github.com/Grubbomatic/Crawl-Readiness/issues. Email: crawlreadiness@gmail.com.

MIT license. The server calls live endpoints at crawlreadiness.com — same rate limits and fair-use rules as the web dashboard.