How we calculate your AI readiness score
A 100-point starting score with defined deductions for each signal we can detect from public data. No hidden math, no rank inflation. This page is the authoritative source for our methodology.
The scoring model in one paragraph
Every site starts at 100. We deduct points for each check that fails, weighted by how much that check actually affects whether an AI crawler can access and understand your page. The final score is capped at 0 and 100 — together these signals make up your GEO (generative engine optimization) readiness: how ready your site is to be read and cited by AI answer engines.
High-severity checks take the most: blocked AI search and answer crawlers up to 30 points, a JS-only homepage 20, an indexing block 10. What doesn't move the score at all: crawlers that only gather training data, emerging conventions (llms.txt, agents.json, Content-Signal) and agent readiness (a markdown copy, forms an agent can use). Every report still shows them, under “Going further”.
What the grade means
The score is a number; the grade is what we're willing to say about it. Five grades come straight from the score. Three don't — they exist because a low number can mean very different things, and reporting them all as failure would be wrong.
Restricted by Design and JavaScript-Dependent keep their score and their rank — we don't inflate either one. What changes is the verdict attached to it. A publisher who blocks AI crawlers on purpose is working as configured, not failing; a large web app that renders client-side is making a normal engineering trade-off. Reporting a deliberate choice as an F would be a judgement we have no business making about someone else's website.
Partly Checked is the opposite problem. When a site's server refuses our crawler — a 403, a 429, a timeout — every page-content check is skipped, and skipping deductions pushes the score up. A site can score 90 without us reading a single byte of its homepage. That grade exists so a partial check can never be published as an A. The same block, incidentally, is likely to apply to GPTBot and ClaudeBot as well.
A site we couldn't reach at all gets no grade and no score. We return an error instead of a report, because a check where nothing answered tells you nothing.
Every check, every weight
The table below shows the current version of every check we run, its point weight, and its severity. These values live in lib/checker.js and change only in tagged releases.
What we check: We fetch your robots.txt and test it against 51 known AI crawler user-agents. The score counts the ones that fetch pages for AI search results and answers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Google-Extended and the rest; the full list is below). If more than half of those are Disallowed, this fires.
Why it matters: This is the single biggest thing that stops AI from seeing your site. If ChatGPT-User is blocked, ChatGPT literally cannot fetch your pages when a user asks about you.
What we check: We fetch your homepage HTML without executing JavaScript and measure the text-only content length. If it's below thresholds (300 chars, or 600 with SPA signals like <div id="root">), this fires.
Why it matters: Most AI crawlers don't execute JavaScript. A single-page app that renders content client-side looks like an empty shell to them.
What we check: We deduct 5 points per blocked crawler that the score counts, capped at 20. Once more than 20% of them are blocked it is a flat 20, and above 50% the majority check above applies instead (30). Blocking more crawlers never costs fewer points. Crawlers that only gather training data are listed but not counted.
Why it matters: Each of these crawlers fetches pages for an AI search engine or assistant, so every one you block is an answer engine that can't cite you.
What we check: We parse the homepage <meta name="robots"> tag for noindex, none, or noai directives.
Why it matters: A page-level block overrides a permissive robots.txt. This is a common accidental self-inflicted wound.
What we check: We read the X-Robots-Tag HTTP response header on the homepage for noindex, none, or noai. Scored separately from the meta tag, so a site sending both loses 15.
Why it matters: A header-level block applies even when the HTML looks permissive, and it is easy to miss because it never appears in the page source.
What we check: We scan the homepage for <script type="application/ld+json"> blocks. This check is presence-only — it does not validate the markup. The free Schema Validator tool does that.
Why it matters: Structured data gives AI systems clean facts about your site (name, type, products, reviews) instead of forcing them to infer from raw HTML.
What we check: We check for a <meta name="description"> tag on the homepage with at least a few characters of content.
Why it matters: AI systems often use the meta description as the first summary of what a page is about. Missing it means AI picks a random sentence.
What we check: We look for a Sitemap: directive in robots.txt pointing to an XML sitemap.
Why it matters: Crawlers discover pages via internal links + sitemaps. Without one they may miss pages that aren't linked from the homepage.
What we check: We look for an og:title or og:description tag on the homepage. Either one is enough.
Why it matters: AI systems and link previews lean on these when summarizing or citing a page.
Going further: shown, never scored
Every report lists these under “Going further”. They're worth knowing about and often cheap to do, but AI systems don't depend on them yet, so none of them adds or takes away a point.
What we check: Crawlers whose makers document them as gathering training data only (GPTBot, ClaudeBot, Applebot-Extended, Common Crawl's CCBot and others), plus Google's research crawlers, an owner-only crawler and retired tokens. The full list is below.
Why it isn't scored: Blocking these keeps your content out of future AI models without keeping you out of AI search or assistant answers. A valid choice, so it doesn't cost points.
What we check: We fetch /llms.txt and count it only when the answer is a text file, not a web page standing in for a missing one, and we note whether its first line is the Markdown heading the convention expects.
Why it isn't scored: An emerging convention. Few AI crawlers request it yet (an Ahrefs study of 137,000 sites in 2026 found 97% of llms.txt files got no requests at all), and Google's guidance says no special optimizations are needed for AI Overviews. Cheap to add, but not something AI depends on.
What we check: We fetch /.well-known/agents.json to check for AI-facing endpoint declarations.
Why it isn't scored: An emerging, loosely specified convention. No major AI crawler is documented as reading it yet.
What we check: We parse robots.txt for a Content-Signal line declaring AI training / search / input preferences.
Why it isn't scored: From Cloudflare's Content Signals Policy: states your AI preferences in one line. Useful, and optional.
What we check: We request your homepage a second time with Accept: text/markdown and check whether the answer is markdown. We also note a <link rel="alternate" type="text/markdown"> tag pointing at a markdown copy.
Why it isn't scored: A markdown copy strips menus and scripts, so an agent reads your content with about 80% fewer tokens (Cloudflare's figure). Few sites serve one yet.
What we check: For each form in the homepage HTML: whether every field has a label an agent can read (a <label>, aria-label, aria-labelledby or title; a placeholder alone doesn't count), and whether the form declares itself as a WebMCP tool (toolname and tooldescription attributes).
Why it isn't scored: Browser agents that fill in forms read the same labels screen readers use. WebMCP is a Chrome trial that lets a form become a tool an agent calls directly; browsers without it ignore the tags.
Which AI crawlers count
We test all 51 crawlers. The score counts the 35 that fetch pages for AI search results or answers, including ones whose makers document a mixed use or none at all. Google-Extended counts because Google says it also controls grounding in the Gemini apps, not just training; meta-externalagent counts because Meta says it also indexes content for its products. The 16 below are listed in every report but not counted. Purposes come from each vendor's own documentation where it exists (OpenAI, Anthropic, Google).
GPTBot training only · ClaudeBot training only · Claude-Web retired token · anthropic-ai retired token · GoogleOther research crawl · Google-CloudVertexBot only for the site owner · GoogleOther-Image research crawl · GoogleOther-Video research crawl · FacebookBot training only · Applebot-Extended training only · cohere-training-data-crawler training only · CCBot training only · Bytespider training only · AI2Bot training only · AI2Bot-Dolma training only · NeevaBot retired token
OAI-SearchBot · ChatGPT-User · Claude-User · Claude-SearchBot · Google-Extended · Gemini-Deep-Research · Google-NotebookLM · PerplexityBot · Perplexity-User · GrokBot · Grok-DeepSearch · xAI-Grok · Copilot · CopilotNative · meta-externalagent · meta-externalfetcher · Amazonbot · PetalBot · YandexAdditional · YandexAdditionalBot · DeepSeekBot · MistralAI-User · cohere-ai · DuckAssistBot · Diffbot · PhindBot · YouBot · Kagibot · AndiBot · Timpibot · SiderAI · HuggingFace-Bot · Quora-Bot · GensparkBot · Elicit-Bot
What we deliberately don't measure
Being honest about what the score doesn't tell you is as important as showing what it does.
Whether AI actually cites you
The check tells you whether AI can technically read your site. It does NOT tell you whether ChatGPT, Claude, or Perplexity actually mentions you when customers ask relevant questions. That's what our LLM Monitor measures — it queries real AI providers with real prompts and tracks mention rate + brand position over time.
Ranking or authority
AI systems weigh domain authority, backlinks, freshness, and query-specific relevance. The check does not measure those. A perfect 100 doesn't guarantee AI citations — it just removes the technical obstacles.
Content quality
We don't judge whether your content is good, useful, or accurate. We check whether it's readable by machines.
What the paid tiers add — and why
The free check tells you if AI can read your site. What paid tiers add is ongoing measurement: whether AI actually cites you, tracked over time, across multiple providers. That's the difference between "are the doors unlocked" and "are customers walking in."
Free
- All 6 free web tools (llms.txt, schema, robots.txt — generators and validators)
- Chrome extension (checks count toward your quota)
- Public shareable report card
Best for: Trying the tool, checking one or two sites you own
Pro
- Weekly OR daily auto-schedule (Monitor runs on its own)
- Per-provider trend charts + share-of-voice history
- 5 API keys for programmatic checks
- Per-crawler detail and fixes inside the Chrome extension
- Priority support
Best for: Solo consultants and indie hackers who monitor 1–5 client or their own site's AI visibility
Agency
- Everything in Pro
- 20 competitors tracked per project (share-of-voice vs specific rivals)
- Bulk check: up to 100 sites scored in one job, results as CSV
- Client workspaces: everything filed by client, from one login
- The largest API quota — 5,000 checks / month
Best for: Marketing / SEO agencies managing 10–30 client sites with recurring AI-visibility reporting
How the LLM Monitor measures AI answers
The Monitor asks each AI system your prompts and checks whether your brand appears in the answer. Not all engines answer the same way, and we're explicit about the difference:
- Live-search engines — Perplexity and Google AI Overviews. Queried against real-time search results (Perplexity Sonar; Google AI Overviews via SerpAPI). These reflect what the web says right now.
- Model-knowledge engines — ChatGPT and Claude. Queried through each provider's API (gpt-4o-mini, Claude Haiku), which answer from model training rather than a live web session, so they won't reflect very recent web changes. Their answers also vary from call to call, so on Pro and Agency each prompt is asked three times and the mention rate is reported as the mean with the range across samples — the shaded band on the trend chart. Free asks once. Live-search engines are asked once per prompt: their answers are anchored by search results, and each call costs several times more.
The two live-search engines also return the pages they drew on. The Monitor keeps them with each answer and tallies, per run, which sites were cited and how often, including whether yours was one of them. ChatGPT and Claude, asked from model knowledge, cite nothing, so they are left out of that tally.
Because we measure the same way every run, the trend and share-of-voice vs competitors stay reliable regardless of engine type. Absolute answers on the two model-knowledge engines can differ from a specific user's app session (model version, custom instructions, browsing state). Most competitors don't disclose which engines are live vs model-based — we do.
Known limitations
- We test robots.txt directives, not enforcement. A crawler could ignore your robots.txt (well-behaved ones don't).
- We fetch as ourselves. The check identifies itself as
CrawlReadinessBot/1.0and evaluates your published rules for each AI crawler — it does not impersonate them. Some sites serve different content by user-agent, IP range or fingerprint, which a single vantage point can't see. The separate Content Parity Checker does request your page as a browser and as GPTBot, ClaudeBot and PerplexityBot, to compare what each is served. - The homepage is one page. Structural signals (robots.txt, sitemap, schema types) apply site-wide, but content signals (readable text, meta, forms, the markdown copy) are checked on the homepage only.
- Forms that JavaScript adds after the page loads aren't seen. The agent-forms check reads the HTML the server sends, like most AI crawlers do.
- Google AI Overviews query rendering is emulated via SerpAPI in the Monitor. We can't query Google's AI Mode directly; SerpAPI is the closest proxy available. Google shows no AI Overview for many questions; those are left out of the Google figures and your mention rate rather than counted as a miss. Until September 26, 2026 the Monitor read Google's regular search snippets for those questions instead, so earlier Google figures aren't directly comparable.
- Sentiment classification uses Claude Haiku per response. If that call fails or our Anthropic quota is exhausted, we fall back to a simpler keyword-based classifier for that answer. The fallback is not currently labelled in the UI, so treat sentiment as directional rather than exact.
Data handling
What data do you store when I run a check?
The URL you submitted, the score, per-check results, and (if you're signed in) a link to your account. We do NOT store the full HTML content of your pages — only extracted signals (robots.txt presence, meta tags, structured-data blocks).
How long do you retain check data?
Every check, signed in or not, produces a report card at an unlisted link that is retained indefinitely, and the checked domain can appear on the public leaderboard. Reports show the site that was checked, never who checked it. Signed-in checks are additionally listed in your dashboard history and deleted on request (email support@crawlreadiness.com). Monitor query results: retained per your subscription; deleted 30 days after account cancellation.
Are shareable reports public?
Yes — the /report/[id] URL is public and can be shared. If you don't want a public report for a domain you own, email support@crawlreadiness.com and we'll remove it within 5 business days.
Where is the service hosted?
Vercel (application) + Supabase (database + auth). Tundrastone is an Ontario, Canada business.
How do I report a security issue?
Email support@crawlreadiness.com with 'security' in the subject. We aim to acknowledge within 48 hours.
Changes to this methodology
When we change how the score is calculated, we'll note it here with the date and the reason. The scoring model version is currently v2.0 (29 September 2026).
- Blocked crawlers count only when they fetch pages for AI search or answers. Blocking a training-only crawler is a choice about training, not about being found, so it's listed but no longer costs points.
- llms.txt (was 5 points), agents.json (3) and Content-Signal (2) no longer count. They're emerging conventions AI systems don't depend on yet, and they're still shown in every report.
- New, not scored: whether the homepage serves a markdown copy to AI agents, and whether its forms are labelled and tagged for agents.
- Most changed scores went up: sites that block only training crawlers, or lack the three files. One case can go down: the “more than half blocked” band now applies to the crawlers that count, so a site that blocks most AI search crawlers while leaving the training ones open can land in it. Shared reports and leaderboard rows keep the score they were given until the site is checked again, and watched sites re-baseline on their next check instead of alerting.
Questions or think we're weighing something wrong? Email support@crawlreadiness.com.