Free · works in Claude, Cursor, Cline & Windsurf

Crawl Readiness MCP Server

AI SEO tools inside Claude, Cursor, Cline, and any MCP-compatible client. Audit a site, then generate the files that fix it — without leaving the conversation.

What you can ask for

Once it's installed, these are real requests your assistant can carry out — no tab-switching, no copy-pasting URLs into a form.

“Is my brand showing up when people ask AI about my category?”

Share of voice, average position, per-provider mention rate and a 12-run trend — read straight out of your LLM Monitor without opening a dashboard.

“Can ChatGPT and Claude actually read acme-inc.com?”

Scores the site against 50+ AI crawlers and hands back a ranked fix list, with the per-crawler robots.txt verdict for each.

“Generate an llms.txt for this site and write it to /public.”

Crawls the sitemap, groups the pages, and produces the finished file — your assistant can save it into the repo it already has open.

“What do AI crawlers see on this page that humans don't?”

Fetches the page as Chrome, GPTBot, ClaudeBot and PerplexityBot in parallel and reports the divergence.

Eight tools in total, hitting our live API at crawlreadiness.com. The four audit tools work anonymously. The three generators and the Monitor read need a free API key — no card, takes a minute.

What is MCP?

Model Context Protocol is Anthropic's open standard for connecting AI assistants to real tools. Any MCP-compatible client — Claude Desktop, Cursor, Cline, Windsurf, Continue — can call an MCP server's tools directly in the conversation. Instead of copy-pasting between your editor and a browser tab, your AI does the work.

Install for Claude Desktop

Open your config file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add this to the mcpServers block:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"]
    }
  }
}

Restart Claude Desktop. You'll see "crawl-readiness" listed in the tools panel with the four audit tools active.

Install for Cursor, Cline, Windsurf, Continue

Same shape — add crawl-readiness to your MCP servers list with npx -y crawl-readiness-mcp as the command. In Cursor: Settings → Features → Model Context Protocol → Add new global MCP server.

Unlock the generator tools

The four audit tools work anonymously. The three generators (generate_llms_txt, generate_robots_txt, generate_schema) and get_monitor_trend need an API key — free at crawlreadiness.com/dashboard (50 checks/month, no credit card).

Add the key to the same config:

{
  "mcpServers": {
    "crawl-readiness": {
      "command": "npx",
      "args": ["-y", "crawl-readiness-mcp"],
      "env": {
        "CRAWL_READINESS_API_KEY": "cr_live_xxxxxxxxxxxx"
      }
    }
  }
}

Restart the client. All eight tools are now callable.

The eight tools

get_monitor_trendAPI key

Read-only view of your LLM Monitor data: share of voice, average position, per-provider mention rate, competitor breakdown and a 12-run trend. Call it with no argument to list your tracked brands, or pass a brand name or project id for the detail. Never triggers a run, so it costs nothing.

check_ai_readinessFree

Score a site 0–100 against 50+ AI crawlers (ChatGPT, Claude, Perplexity, Google AI). Returns per-crawler robots.txt status, homepage signals, structured-data flags, and a prioritized fix list.

validate_schemaFree

Audit every JSON-LD block on a URL against 20+ common types (Article, Organization, Product, LocalBusiness, FAQPage, etc.). Reports required-field errors, recommended-field warnings, and type-specific gotchas.

validate_robotsFree

Line-by-line audit of a robots.txt file. Detects syntax errors, orphan Allow/Disallow lines, unofficial Noindex, wildcard traps, and summarizes AI-bot coverage across 50+ known AI crawlers.

check_content_parityFree

Fetches a page as Chrome, GPTBot, ClaudeBot, and PerplexityBot in parallel. Reports word-overlap %, title/description differences, and warnings about JS-only shells, cloaking, or edge-based bot blocking.

generate_llms_txtAPI key

Crawl a site's sitemap, group pages by section, pull titles and descriptions, and produce a properly formatted llms.txt plus a companion robots.txt snippet — ready to write into /public/.

generate_robots_txtAPI key

AI-crawler-aware robots.txt with four presets: allow-all (public business), search-only, recommended (allow assistants that cite, block training-only bots), and block-all.

generate_schemaAPI key

Detects site name, logo, social profiles, contact info, and article metadata, then produces Organization, WebSite, and (when applicable) Article JSON-LD — each as a ready-to-paste <script type="application/ld+json"> block.

Example prompts

Once installed, try any of these in your AI client:

“Is my portfolio site AI-readable? URL: yourdomain.com”
“Check if ChatGPT can see acme-inc.com and walk me through the fixes.”
“Validate the JSON-LD on stripe.com — is anything missing?”
“Compare what humans see vs what AI crawlers see on producthunt.com.”
“Generate an llms.txt for my site and save it to /public/llms.txt.”

Your AI can chain tools automatically — audit a site, identify issues, generate the fix files, and write them straight into your project.

Environment variables

Variable	Purpose	Default
`CRAWL_READINESS_API_KEY`	API key for generator tools	none
`CRAWL_READINESS_API_BASE`	Override API base URL (for testing)	https://www.crawlreadiness.com

Source & support

Source code: github.com/Grubbomatic/crawl-readiness-mcp. Issues: github.com/Grubbomatic/crawl-readiness-mcp/issues. Email: support@crawlreadiness.com.

MIT license. The server calls live endpoints at crawlreadiness.com — same rate limits and fair-use rules as the web dashboard.