Why this matters
Every month, more of your customers ask AI assistants questions instead of Googling. When they ask "what's the best marketing platform for a solo consultant", the AI has to decide who to mention. If your site is invisible to ChatGPT, Claude, or Perplexity, you're invisible to your buyer in that moment.
Being invisible is more common than you'd think. Sites accidentally block AI crawlers all the time — through Cloudflare bot fights, misconfigured robots.txt, JavaScript-only rendering, or plain-old paywalls. This guide walks through the five checks you should run on every site.
Check 1: Is your robots.txt blocking AI bots?
Your robots.txt file (at yourdomain.com/robots.txt) tells crawlers which parts of your site they can access. Many sites — especially those on Cloudflare with the "AI Scrapers and Crawlers" feature turned on — block AI bots by default without the owner realizing.
Look for lines like these in your robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /If you see them and you want AI to be able to read your site, remove the Disallow lines or change them to Allow: /.
Also worth checking: Google's Google-Extended user-agent controls Gemini and AI Overview training. It's separate from Googlebot. Blocking one doesn't block the other.
Quick check: Paste your URL into our free robots.txt Validator and it'll tell you exactly which of the 50+ tracked AI bots are being blocked. No signup required.
Check 2: Does your homepage need JavaScript to show content?
Most AI crawlers don't execute JavaScript. If your homepage relies on client-side rendering (React, Vue, or a single-page app without server-side rendering), the AI sees a nearly-empty HTML shell.
Test this yourself: view your page's source in Chrome (Ctrl/Cmd + U). If the <body> is nearly empty and the real content is loaded by JavaScript, AI crawlers will see almost nothing.
Fixes: server-render your homepage (Next.js's getServerSideProps, Nuxt's SSR, or Astro), or use a prerendering service like Prerender.io. This is often the single biggest fix — worth 10-20 points on our AI Readiness Score.
Quick check: Our Content Parity Checker fetches your page four times — as a normal browser, as GPTBot, as ClaudeBot, and as PerplexityBot — and compares what each sees. If the crawlers get a nearly-empty page while you get a rich one, this is your problem.
Check 3: Do you have structured data (JSON-LD)?
Structured data is machine-readable metadata that describes your page. Types includeOrganization, WebSite, Article,Product, FAQPage, and dozens more. AI systems use it to extract clean facts about you.
Without structured data, an AI assistant answering "what does Acme Corp do?" has to guess your name, description, socials, and contact info from your HTML. With structured data, it just reads them from your <script type="application/ld+json"> block.
Minimum viable structured data for every site:
- Organization — your name, logo, socials (add sitewide)
- WebSite — with SearchAction if you have a search box
- Article — per blog post
Quick fix: Our free Schema Wizard scans your site, detects what it can (name, logo, socials, contact info), and generates all three schemas as copy-paste-ready<script> tags.
Check 4: Is your site behind a CDN that blocks bots?
Cloudflare, AWS WAF, and similar services often block "unknown" or "AI" user-agents at the edge — before the request even reaches your server. This won't show up in your robots.txt because the block is at a different layer.
On Cloudflare specifically:
- Dashboard → Security → Bots → check "Block AI Scrapers and Crawlers". If it's on, you're blocking most AI training crawlers by default.
- Dashboard → Security → WAF → Managed Rules → check if "Bot Fight Mode"is on. Bot Fight Mode blocks anything Cloudflare thinks isn't a mainstream browser.
Quick check: Our Content Parity Checker will surface HTTP 403 or 404 responses that only affect AI crawlers — a classic edge-level block symptom.
Check 5: Does your meta and OG data actually describe you?
AI systems lean heavily on:
<title>— what the page is about<meta name="description">— the one-line summary<meta property="og:image">— the image used when sharing<meta property="og:site_name">— your brand name
These aren't optional. A missing meta description costs you AI citations the same way it costs you rich Google snippets.
Run everything at once
We built a free tool that runs all five checks against your site in about 10 seconds and gives you a 0-100 score with specific fixes for each issue: the Crawl Readiness AI Readiness Checker.
It checks robots.txt against 50+ AI crawler user-agents, tests for structured data, evaluates meta and OG tags, detects JavaScript-only rendering, checks for llms.txt and agents.json, and produces a shareable report card. No signup needed for the basic check.
What to fix first
If your score is low, prioritize in this order:
- Unblock crawlers in robots.txt and at the CDN — no other fix matters if AI can't reach you at all.
- Fix JS-only rendering if you have it — this makes your content visible for the first time.
- Add structured data — takes 5 minutes with our Schema Wizard, huge upside for how you're cited.
- Add llms.txt — cheap, forward-looking, increasingly used.
- Add meta descriptions and OG tags — table stakes for both SEO and AI.
Everything above is free. The paid tier of Crawl Readiness ($29/mo Pro) adds LLM answer monitoring — actual tracking of whether ChatGPT and Claude mention your brand for target prompts, with a weekly trend line. That's the ROI story you can bring to your boss.