machinereadiness.com
Open to AI crawlers, with the metadata to be understood.
Summary
machinereadiness.com is rated A AI Ready with a score of 100 out of 100 on the site-level check. 4 pages were measured: 4 readable without JavaScript, 3 with structured data, 0 with a markdown copy for agents. The access policy reads "Open to AI". Your robots.txt says "open to ai", your other signals agree, and the server does what it says for the 4 crawlers we tested. 35 of 35 AI search and answer crawlers can read the site. 1 improvement is listed in the action plan, starting with 1 of the 4 pages checked have no structured data.
What this report is: every page we could find measured as a crawler receives it, the site-level signals and the access policy verified at the server, the five files that fix what is wrong written for your platform, one Answer Analysis on live AI search, and a weekly re-check for thirty days. Nothing here promises a mention or a rank; it says what machines can read, and what your site tells them they may do.
Access policy
What the site says machines may do with it, declared by robots.txt, and whether the server does what the rules say. Your public statement reads "AI search: yes · AI training: yes".
Your robots.txt says "open to ai", your other signals agree, and the server does what it says for the 4 crawlers we tested.
| Crawler | Purpose | robots.txt says | Server answered | Outcome |
|---|---|---|---|---|
| GPTBot | training | allow | HTTP 200 | served, as allowed |
| OAI-SearchBot | search | allow | HTTP 200 | served, as allowed |
| ClaudeBot | training | allow | HTTP 200 | served, as allowed |
| PerplexityBot | search | allow | HTTP 200 | served, as allowed |
- Content-SignalContent-Signal agrees with the rules
- TDM reservationno TDM reservation, and none expected for a policy that allows training
- Page-level directivesno noai directive on the homepage
Every page
Each page as an AI crawler receives it, without running JavaScript: whether the content is in the HTML, whether it carries structured data and a description, whether its form fields are labelled for agents, and whether the server offers a markdown copy when asked for one. Tokens are what an AI reads to take the page in, at about four characters per token.
| Page | Readable | JSON-LD | Description | Forms | Markdown | Tokens |
|---|---|---|---|---|---|---|
| / Machine Readiness — readable and usable by machines, proven | ✓ | ✓ | ✓ | · | ✗ | 7,847 |
| /standard The Machine Readiness Standard, version 1.0 · Machine Readiness | ✓ | ✓ | ✓ | · | ✗ | 8,462 |
| /crawlers The AI crawler registry · Machine Readiness | ✓ | ✗ | ✓ | · | ✗ | 12,176 |
| /permission The permission API: may I use this page for this? · Machine Readiness | ✓ | ✓ | ✓ | · | ✗ | 9,265 |
Readiness profile
The four areas the score counts on the homepage: how easily AI systems can reach, read and understand the site.
Site-level signals
- robots.txt reachableThe file was read and its rules applied per crawler.
- Homepage readable without JavaScriptThe HTML carries the content; crawlers that do not run JavaScript still see it.
- Structured data (JSON-LD)Machine-readable facts about the organisation or content are present.
- Meta descriptionA summary AI systems can quote is present.
- Open Graph tagsLink previews and citations have a title and description to use.
- No page-level AI or index blockNeither the robots meta tag nor the X-Robots-Tag header blocks indexing.
- Sitemap listed in robots.txtCrawlers are told where the full page list lives.
- llms.txt · not scoredA Markdown summary for AI systems is published at the site root.
- agents.json · not scoredPublished under /.well-known.
- Content-Signal directive · not scoredDeclares search=yes, ai-input=yes, ai-train=yes.
- RSS or Atom feed · informationalNo feed advertised.
- Markdown copy for AI agents · not scoredAsking the homepage for markdown (Accept: text/markdown) returns HTML. Few sites offer a markdown copy yet.
AI crawler access
What the site's robots.txt says to each AI vendor's crawlers. A blocked crawler cannot fetch pages for that vendor's answers. Every AI search and answer crawler checked is allowed.
| Vendor | Feeds | Status | Crawlers |
|---|---|---|---|
| OpenAI | ChatGPT answers and ChatGPT search | Allowed | ✓ GPTBot✓ OAI-SearchBot✓ ChatGPT-User |
| Anthropic | Claude answers and Claude's web search | Allowed | ✓ ClaudeBot✓ Claude-User✓ Claude-SearchBot✓ Claude-Web✓ anthropic-ai |
| Google AI | Gemini and Google's AI features (Search itself uses Googlebot) | Allowed | ✓ Google-Extended✓ GoogleOther✓ Gemini-Deep-Research✓ Google-NotebookLM✓ Google-CloudVertexBot✓ GoogleOther-Image✓ GoogleOther-Video |
| Perplexity | Perplexity answers | Allowed | ✓ PerplexityBot✓ Perplexity-User |
| Microsoft Copilot | Copilot answers | Allowed | ✓ Copilot✓ CopilotNative |
| Meta AI | Meta AI across Facebook, Instagram and WhatsApp | Allowed | ✓ meta-externalagent✓ meta-externalfetcher✓ FacebookBot |
| Apple Intelligence | Siri and Apple Intelligence | Allowed | ✓ Applebot-Extended |
| xAI / Grok | Grok answers | Allowed | ✓ GrokBot✓ Grok-DeepSearch✓ xAI-Grok |
| Amazon | Allowed | ✓ Amazonbot | |
| Mistral | Allowed | ✓ MistralAI-User | |
| DeepSeek | Allowed | ✓ DeepSeekBot | |
| 19 other vendors | Allen AI, Andi, Bytedance, Cohere, Common Crawl, Diffbot, DuckDuckGo AI, Elicit, Genspark, Huawei, HuggingFace, Kagi, Neeva, Phind, Quora / Poe, Sider, Timpi, Yandex AI, You.com | Allowed | ✓ all 22 crawlers |
The files, ready to install
The files below were written for machinereadiness.com from what the audit found. None of them changes your policy: the audit read the stance your robots.txt states today (“Open to AI”: AI search: yes, AI training: yes) and wrote each file to say the same thing, for every AI crawler we track, in the place AI systems look for it. To take a different stance, the robots.txt generator writes all five from one choice.
The site is served by Vercel, so all five files are plain files in the project and a deploy publishes them. All five are plain files, so this is a short job for whoever looks after the site; the box at the end of this section is written for them.
robots.txt · /robots.txt
What it does. Tells each AI crawler, by name, whether it may read the site. Most crawlers check it before every visit.
Why it matches your policy. Your existing rules stay exactly as they are; none of them names an AI crawler. Below them, an AI section applies your policy to all 51 AI crawlers we track: 51 allowed, 0 blocked. The Content-Signal line under the rules states the same policy in one line, for crawlers that read it. Checked before it was handed to you: the 1 path rule under your "User-agent: *" and every path your file names (1 in all) give each of the 51 AI crawlers the same answer or a stricter one with this file; nothing you closed is opened.
User-agent: * Allow: / # The standard's own site is open to AI search, answers, agents and training. Sitemap: https://machinereadiness.com/sitemap.xml # ---- AI crawler policy (managed by Crawl Readiness) ---- # Generated at https://www.crawlreadiness.com/tools/robots-txt # 51 AI crawlers are allowed by your "User-agent: *" rules and follow them, so they are not listed here. # Content-Signal declares your training / search / AI-input preferences. User-agent: * Content-Signal: search=yes, ai-input=yes Sitemap: https://machinereadiness.com/sitemap.xml # ---- end AI crawler policy ----
How to install it on Vercel
- Replace the site's robots.txt with the file above. Your existing rules are unchanged; the AI section sits below them. On Vercel the root is the project's public folder (public/ in a Next.js project; the static folder of other frameworks); push the change and the next deploy publishes it.
- Check: open https://machinereadiness.com/robots.txt in a browser. The AI crawler section should be there, at the end. Then run the free check at crawlreadiness.com: the crawler table should match the policy.
llms.txt · /llms.txt
What it does. A plain-text introduction to the site for AI assistants: what it is, and the 4 pages that matter most with a line on each, from the sitemap.
Why it matches your policy. It helps an assistant describe the site accurately. It grants nothing and blocks nothing, so it fits any policy.
# Machine Readiness > The Machine Readiness standard: what it means for a website to be readable by AI crawlers, usable by AI agents and verifiably permitted for each use. The crawler registry, the permission check and… This file helps AI assistants understand this site. It lists the pages we consider most useful for machine consumption. Learn more at https://llmstxt.org ## Overview - [Machine Readiness — readable and usable by machines, proven](https://machinereadiness.com/): The Machine Readiness standard: what it means for a website to be readable by AI crawlers, usable by AI agents and verifiably permitted for each use. The crawler registry, the permission check and… ## Standard - [The Machine Readiness Standard, version 1.0 · Machine Readiness](https://machinereadiness.com/standard): What a website must do to be readable by AI crawlers, have its access policy expressed and enforced, be usable by AI agents, and have all of it verified from outside. Four parts, twenty-one checks… ## Crawlers - [The AI crawler registry · Machine Readiness](https://machinereadiness.com/crawlers): Every AI crawler token the Machine Readiness standard knows, by vendor and purpose: search index, AI answers, training, user-requested fetches. Human-readable here, machine-readable at /crawlers.json… ## Permission - [The permission API: may I use this page for this? · Machine Readiness](https://machinereadiness.com/permission): One call: a page and a purpose in; what robots.txt, Content-Signal, Content-Usage, an RSL licence, the TDM reservation, ai.txt and the page's directives say out, with what the server did when asked…
How to install it on Vercel
- Save the text above as a file named llms.txt and put it in the root, next to robots.txt (the robots.txt step says where that is), so it opens at https://machinereadiness.com/llms.txt.
- Check: open https://machinereadiness.com/llms.txt. You should see the text itself: not a “page not found”, not a download. Hosts serve a .txt file as plain text on their own; nothing else is needed.
- Keep it current: when the main pages change, run the free llms.txt generator at crawlreadiness.com again and replace the file.
JSON-LD structured data · the <head> of every page
What it does. The facts about Machine Readiness: who it is, what it is and where things are, in the format search engines and AI systems read without guessing. The Organization and WebSite scripts go on every page; the others are templates for one kind of page.
Why it matches your policy. It describes the site; it does not grant or refuse access, so it fits any policy.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Machine Readiness",
"url": "https://machinereadiness.com",
"logo": "https://machinereadiness.com/favicon.svg",
"description": "The Machine Readiness standard: what it means for a website to be readable by AI crawlers, usable by AI agents and verifiably permitted for each use. The crawler registry, the permission check and the public record, by the makers of Crawl Readiness.",
"contactPoint": {
"@type": "ContactPoint",
"contactType": "customer service",
"email": "support@crawlreadiness.com"
}
}
</script><script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "WebSite",
"name": "Machine Readiness",
"url": "https://machinereadiness.com",
"description": "The Machine Readiness standard: what it means for a website to be readable by AI crawlers, usable by AI agents and verifiably permitted for each use. The crawler registry, the permission check and the public record, by the makers of Crawl Readiness."
}
</script><script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{
"@type": "ListItem",
"position": 1,
"name": "Home",
"item": "https://machinereadiness.com/"
},
{
"@type": "ListItem",
"position": 2,
"name": "Category",
"item": "https://machinereadiness.com/category/"
},
{
"@type": "ListItem",
"position": 3,
"name": "Current page",
"item": "https://machinereadiness.com/category/current-page"
}
]
}
</script><script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is your first question?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Your answer goes here. Keep it concise but complete — AI assistants may quote it verbatim."
}
},
{
"@type": "Question",
"name": "What is your second question?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Your second answer goes here."
}
}
]
}
</script>How to install it on Vercel
- Each script goes between <head> and </head> of the page's HTML. The Organization and WebSite scripts belong on every page, so put them in the shared layout or header template (a developer will know the file: layout.tsx, _document.js, base.html, header.php).
- The Article script is a template: use it only on article pages, filled with each page's own title, dates and author.
- Check: open the homepage, press Ctrl+U to view the page source, and search for application/ld+json; or paste the address into Google's Rich Results Test.
tdmrep.json · /.well-known/tdmrep.json
What it does. The machine-readable opt-out from text and data mining that EU law recognises: the file a crawler looks for when it wants the answer about training in one place.
Why it matches your policy. Not written for you. Your robots.txt does not reserve AI training, and a file that said “not reserved” would waive the EU opt-out from text and data mining in writing. That is your decision, not the audit's. If you decide to reserve training, publish the two lines below.
[
{
"location": "/",
"tdm-reservation": 1
}
]How to install it on Vercel
- Create a folder named .well-known (with the leading dot) in the root, next to robots.txt, and save tdmrep.json inside it, so it opens at https://machinereadiness.com/.well-known/tdmrep.json. On Vercel the root is the project's public folder (public/ in a Next.js project; the static folder of other frameworks); push the change and the next deploy publishes it.
- Check: open https://machinereadiness.com/.well-known/tdmrep.json. You should see the two-line JSON, not a “page not found”.
agents.json · /.well-known/agents.json
What it does. A one-file introduction for AI agents: the site's name, what it does and where the summary lives.
Why it matches your policy. It describes; it does not grant or refuse access, so it fits any policy. Extend it with your API or your forms when you want agents to use the site, not only read it.
{
"version": "1.0",
"name": "Machine Readiness",
"url": "https://machinereadiness.com",
"description": "The Machine Readiness standard: what it means for a website to be readable by AI crawlers, usable by AI agents and verifiably permitted for each use. The crawler registry, the permission check and the public record, by the makers of Crawl Readiness.",
"documentation": "https://machinereadiness.com/llms.txt"
}How to install it on Vercel
- Save agents.json in the same .well-known folder, so it opens at https://machinereadiness.com/.well-known/agents.json.
- Check: open https://machinereadiness.com/.well-known/agents.json. You should see the JSON.
- Extend it with your API or your forms when you want agents to use the site, not only read it.
For whoever looks after the site
If someone else maintains machinereadiness.com, send them this section; it is a short job for them. The whole of it:
- Replace /robots.txt with the version in this report: the existing rules stay, the AI section is added below them.
- Add /llms.txt, plain text, from this report.
- Add the Organization and WebSite JSON-LD scripts from this report to the <head> of every page; the Article script only on article pages.
- Add /.well-known/agents.json from this report.
Answer Analysis
10 of your customers' questions, put to live AI search (Perplexity and Google AI Overviews) on October 28, 2026: which answers named or cited machinereadiness.com, and which pages the answers drew on instead. An answer counts when it says “Machine Readiness” or “machinereadiness.com”, or cites a page on machinereadiness.com. The questions were suggested from what the site says it does. Open the analysis →
| Question | Engine | Named or cited you | Pages the answer used |
|---|---|---|---|
| What does it take for a website to be readable by AI crawlers? | Perplexity | no | |
| Google AI Overviews | no | ||
| How can I make my website usable by AI agents? | Perplexity | no | |
| Google AI Overviews | no | ||
| What is the best standard for AI crawler readiness? | Perplexity | no | |
| Google AI Overviews | no | ||
| How do I verify whether AI crawlers can read my website? | Perplexity | no | |
| Google AI Overviews | no | ||
| How can websites prove they are permitted for AI use? | Perplexity | no | |
| Google AI Overviews | no | ||
| What tools check website permissions for AI crawlers? | Perplexity | no | |
| Google AI Overviews | no | ||
| What is the difference between AI-readable and AI-usable websites? | Perplexity | no | |
| Google AI Overviews | no | ||
| How do I prepare my website for AI search and agents? | Perplexity | no | |
| Google AI Overviews | no | ||
| Which standards help websites work reliably with AI systems? | Perplexity | no | |
| Google AI Overviews | no | ||
| How can I keep a public record of AI crawler permissions? | Perplexity | no | |
| Google AI Overviews | no |
What the cited sites publish that machinereadiness.com does not
The sites the answers used most, checked the same way as yours. The signals listed are the ones they have in place and your site lacks.
| Site | Cited | Score | They have, you don't |
|---|---|---|---|
| youtube.com | 8× | 70 · JavaScript-Dependent | nothing you lack |
| reddit.com | 6× | 65 · Restricted by Design | nothing you lack |
| amicited.com | 4× | 100 · AI Ready | Homepage forms labelled for AI agents |
| searchengineland.com | 4× | 95 · Partly Checked | nothing you lack |
| apify.com | 3× | 100 · AI Ready | Markdown copy for AI agents |
| developers.google.com | 3× | 100 · AI Ready | Homepage forms labelled for AI agents |
| facebook.com | 3× | 47 · Restricted by Design | Homepage forms labelled for AI agents |
| linkedin.com | 3× | 57 · JavaScript-Dependent | nothing you lack |
Action plan
In priority order: the site-level findings from the homepage check, then what the page-by-page measurement adds. Each item names the finding and the change that resolves it.
- 11 of the 4 pages checked have no structured datamedium
Add JSON-LD to each page. The Organization and WebSite scripts in the files section go on every page; articles, products and FAQs get a block of their own type on their own page.
What happens next
- machinereadiness.com is re-checked every week until November 27, 2026. You get an email the day its verdict changes, with the findings that moved, and when its server or robots.txt starts treating one of the AI crawlers we test differently, once a second check confirms it. Install the files first, then watch the re-checks pick the changes up.
- The Answer Analysis lives in your dashboard, where the questions can be edited; the Monitor plan runs it monthly.
- The site-level check has its own shareable page: www.crawlreadiness.com/report/pdma7xu4.
- Questions about a finding, or a page we should look at again: reply to the email this report came with.
How this was measured
Every page was fetched once as a crawler would fetch it, without running JavaScript. Crawler access is read from the site's robots.txt for the AI crawlers the standard lists; enforcement is tested by fetching the homepage as four of them and comparing the server's answer with the rules. The markdown copy is tested by asking each page for text/markdown. The Answer Analysis puts each question to live AI search once and records the pages each answer cites; the cited sites are checked the same way as yours. Methodology in full →