How we calculate your AI readiness score

The scoring model in one paragraph

Every site starts at 100. We deduct points for each check that fails, weighted by how much that check actually affects whether an AI crawler can access and understand your page. The final score is capped at 0 and 100. It measures how readable your site is for the AI crawlers you allow: whether they may fetch it, and whether what they fetch can be read and understood. A crawler you block on purpose still costs points, so read the verdict beside the number: a deliberate block is reported as working as intended.

High-severity checks take the most: blocked AI search and answer crawlers up to 30 points, a JS-only homepage 20, an indexing block 10. What doesn't move the score at all: crawlers that only gather training data, emerging conventions (llms.txt, agents.json, Content-Signal) and agent readiness (a markdown copy, forms an agent can use). Every report still shows them, under “Going further”.

What the grade means

The score is a number; the grade is what we're willing to say about it. Five grades come straight from the score. Three don't — they exist because a low number can mean very different things, and reporting them all as failure would be wrong.

Restricted by Design and JavaScript-Dependent keep their score and their rank — we don't inflate either one. What changes is the verdict attached to it. A publisher who blocks AI crawlers on purpose is working as configured, not failing; a large web app that renders client-side is making a normal engineering trade-off. Reporting a deliberate choice as an F would be a judgement we have no business making about someone else's website.

Partly Checked is the opposite problem. When a site's server refuses our crawler — a 403, a 429, a timeout — every page-content check is skipped, and skipping deductions pushes the score up. A site can score 90 without us reading a single byte of its homepage. That grade exists so a partial check can never be published as an A. The same block, incidentally, is likely to apply to GPTBot and ClaudeBot as well.

A site we couldn't reach at all gets no grade and no score. We return an error instead of a report, because a check where nothing answered tells you nothing.

Every check, every weight

The table below shows the current version of every check we run, its point weight, and its severity. These values change only with a new scoring version, and every report records the version that produced it.

Going further: shown, never scored

Every report lists these under “Going further”. They're worth knowing about and often cheap to do, but AI systems don't depend on them yet, so none of them adds or takes away a point.

Which AI crawlers count

We test all 60 crawlers. The score counts the 40 that fetch pages for AI search results or answers, including ones whose makers document a mixed use or none at all. Google-Extended counts because Google says it also controls grounding in the Gemini apps, not just training; meta-externalagent counts because Meta says it also indexes content for its products. The 20 below are listed in every report but not counted. Purposes come from each vendor's own documentation where it exists (OpenAI, Anthropic, Google).

What we deliberately don't measure

Being honest about what the score doesn't tell you is as important as showing what it does.

What the paid options add — and why

The free check tells you how your site presents itself to AI today. The paid options measure more of it and keep it that way. The Machine Readiness Audit measures every page once and writes the fix files for your platform. Monitor re-checks your sites and alerts you the day the verdict or the access policy changes. The Answer Analysis, included with the audit and each plan, shows which pages live AI search used to answer your customers' questions, and what those pages publish that yours don't. It is a diagnostic, not a feed: nothing here promises a mention or a rank.

How the Answer Analysis reads AI answers

An Answer Analysis puts your customers' questions to live AI search, keeps every answer with the pages it drew on, and checks whether your pages and your competitors' appear in it.

• Perplexity is queried through its API (the Sonar model), which searches the web for every prompt and returns its citations.
• Google AI Overviews are read through SerpAPI: the AI Overview Google shows for the query today, with the pages it references. Google shows no Overview for many questions; those are left out of the figures rather than counted as a miss.

Both engines return the pages they drew on. The analysis keeps them with each answer and tallies, per run, which sites the answers used and how often, including whether yours was one of them, and checks those sites with the same readiness check your site gets, so you can see what they publish that you don't.

Until 20 October 2026 the analysis (then called the LLM Monitor) also asked ChatGPT and Claude through their APIs. Those columns answered from model training, not a live web session: they cited nothing, could not know a young brand, and varied from call to call. They were retired on that date; runs recorded before it keep their ChatGPT and Claude rows, labelled as model knowledge, and the trend chart still shows them for those runs.

Known limitations

• robots.txt is a request, not a lock. A crawler that ignores it can still fetch your pages; well-behaved ones don't. Whether your server enforces the policy is tested from outside, for four crawlers (next item).
• We fetch the signals as ourselves, and the homepage status as four crawlers. The check identifies itself as CrawlReadinessBot/1.0 and evaluates your published rules for each AI crawler. Since 20 October 2026 it also requests the homepage as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, status only, for the access-policy panel: that is how it can tell you a crawler you allow is refused, or a crawler you block is served. Some sites serve different content by IP range or fingerprint, which a single vantage point can't see. The separate Content Parity Checker compares the full page each crawler is served.
• The access-policy panel is not scored. It reads which of four policies your robots.txt expresses (open, AI search only, closed, or mixed), checks whether your Content-Signal line, TDM reservation and any noai directive agree with it, and reports whether the server enforced it for the four crawlers above. A deliberate block that holds is reported as working, not as a fault; a block the server does not enforce, or a crawler you allow that the server refuses, is listed in the fixes as an extra.
• The homepage is one page. Structural signals (robots.txt, sitemap, schema types) apply site-wide, but content signals (readable text, meta, forms, the markdown copy) are checked on the homepage only.
• Forms that JavaScript adds after the page loads aren't seen. The agent-forms check reads the HTML the server sends, like most AI crawlers do.
• Google AI Overviews query rendering is emulated via SerpAPI in the Answer Analysis. We can't query Google's AI Mode directly; SerpAPI is the closest proxy available. Google shows no AI Overview for many questions; those are left out of the Google figures rather than counted as a miss. Until September 26, 2026 the analysis read Google's regular search snippets for those questions instead, so earlier Google figures aren't directly comparable.
• Sentiment classification uses Claude Haiku per response. If that call fails or our Anthropic quota is exhausted, we fall back to a simpler keyword-based classifier for that answer. The fallback is not currently labelled in the UI, so treat sentiment as directional rather than exact.

Data handling

Changes to this methodology

When we change how the score is calculated, we'll note it here with the date and the reason. The scoring model version is currently v2.0 (29 September 2026).

Questions or think we're weighing something wrong? Email support@crawlreadiness.com.