[← All posts](https://www.crawlreadiness.com/blog)

September 14, 2026 · 10 min read

# How to check if ChatGPT and other AI crawlers can access your website

A step-by-step guide to checking whether ChatGPT, Claude, Perplexity and Google AI can access your site, plus the five most common blockers and how to fix each one.

*Updated 11 October 2026:* the robots.txt advice now matches how the readiness check has scored since 29 September: crawlers that only gather training data, such as GPTBot and ClaudeBot, no longer count against you. The reasoning is in [How to block AI training without disappearing from AI answers](https://www.crawlreadiness.com/blog/block-ai-training-keep-ai-answers).

**Quick answer:** paste your address into the free [Crawl Readiness check](https://www.crawlreadiness.com/). In about ten seconds it tests your robots.txt against 51 AI crawlers (ChatGPT's, Claude's, Perplexity's, Google's and the rest), checks whether your homepage can be read without JavaScript, and looks for structured data and meta tags, then lists what to fix. No signup. To check by hand, these are the five things it looks at.

## Why this matters

Every month, more of your customers ask AI assistants questions instead of Googling. When they ask *"what's the best marketing platform for a solo consultant"*, the AI has to decide who to mention. If your site is invisible to ChatGPT, Claude, or Perplexity, you're invisible to your buyer in that moment.

Being invisible is more common than you'd think. Sites accidentally block AI crawlers all the time — through Cloudflare bot fights, misconfigured robots.txt, JavaScript-only rendering, or plain-old paywalls. This guide walks through the five checks you should run on every site.

## Check 1: Is your robots.txt blocking AI bots?

Your `robots.txt` file (at `yourdomain.com/robots.txt`) tells crawlers which parts of your site they can access. Many sites — especially those on Cloudflare with the "AI Scrapers and Crawlers" feature turned on — block AI bots by default without the owner realizing.

Look for lines like these in your robots.txt. These are the ones that matter, because these crawlers fetch pages for ChatGPT search, Claude and Perplexity answers:

```
User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /
```

If you see them and you *want* AI to cite your site, remove the Disallow lines or change them to `Allow: /`.

Blocks on `GPTBot` or `ClaudeBot` are different: those crawlers only gather training data, so blocking them keeps your content out of future models without taking you out of AI answers. It's a valid choice, and the readiness check doesn't count it against you. Our [robots.txt guide for AI](https://www.crawlreadiness.com/blog/robots-txt-for-ai) lists which crawler does which job.

Also worth knowing: `Google-Extended` isn't a separate crawler. Googlebot does the fetching, and the Google-Extended token decides whether that content trains Gemini and grounds answers in the Gemini apps. It doesn't affect Google Search or AI Overviews.

**Quick check:** Paste your URL into our free [robots.txt Validator](https://www.crawlreadiness.com/tools/robots-validator) and it'll tell you exactly which of the 50+ tracked AI bots are being blocked. No signup required.

## Check 2: Does your homepage need JavaScript to show content?

Most AI crawlers don't execute JavaScript. If your homepage relies on client-side rendering (React, Vue, or a single-page app without server-side rendering), the AI sees a nearly-empty HTML shell.

Test this yourself: view your page's source in Chrome (Ctrl/Cmd + U). If the <body> is nearly empty and the real content is loaded by JavaScript, AI crawlers will see almost nothing.

Fixes: server-render your homepage (Next.js's `getServerSideProps`, Nuxt's SSR, or Astro), or use a prerendering service like Prerender.io. This is often the single biggest fix — worth 20 points on our AI Readiness Score.

**Quick check:** Our [Content Parity Checker](https://www.crawlreadiness.com/tools/parity) fetches your page four times — as a normal browser, as GPTBot, as ClaudeBot, and as PerplexityBot — and compares what each sees. If the crawlers get a nearly-empty page while you get a rich one, this is your problem.

## Check 3: Do you have structured data (JSON-LD)?

Structured data is machine-readable metadata that describes your page. Types include`Organization`, `WebSite`, `Article`,`Product`, `FAQPage`, and dozens more. AI systems use it to extract clean facts about you.

Without structured data, an AI assistant answering "what does Acme Corp do?" has to guess your name, description, socials, and contact info from your HTML. With structured data, it just reads them from your `<script type="application/ld+json">` block.

Minimum viable structured data for every site:

- **Organization** — your name, logo, socials (add sitewide)
- **WebSite** — with SearchAction if you have a search box
- **Article** — per blog post

**Quick fix:** Our free [Schema Wizard](https://www.crawlreadiness.com/tools/schema-wizard) scans your site, detects what it can (name, logo, socials, contact info), and generates all three schemas as copy-paste-ready`<script>` tags.

## Check 4: Is your site behind a CDN that blocks bots?

Cloudflare, AWS WAF, and similar services often block "unknown" or "AI" user-agents at the edge — before the request even reaches your server. This won't show up in your robots.txt because the block is at a different layer.

On Cloudflare specifically:

- Dashboard → Security → Bots → check **"Block AI Scrapers and Crawlers"**. If it's on, you're blocking most AI training crawlers by default.
- Dashboard → Security → WAF → Managed Rules → check if **"Bot Fight Mode"**is on. Bot Fight Mode blocks anything Cloudflare thinks isn't a mainstream browser.

**Quick check:** Our [Content Parity Checker](https://www.crawlreadiness.com/tools/parity) will surface HTTP 403 or 404 responses that only affect AI crawlers — a classic edge-level block symptom.

## Check 5: Does your meta and OG data actually describe you?

AI systems lean heavily on:

- `<title>` — what the page is about
- `<meta name="description">` — the one-line summary
- `<meta property="og:image">` — the image used when sharing
- `<meta property="og:site_name">` — your brand name

These aren't optional. A missing meta description costs you AI citations the same way it costs you rich Google snippets.

## Run everything at once

We built a free tool that runs all five checks against your site in about 10 seconds and gives you a 0-100 score with specific fixes for each issue: the [Crawl Readiness AI Readiness Checker](https://www.crawlreadiness.com/).

It checks robots.txt against 51 AI crawler user-agents, tests for structured data, evaluates meta and OG tags, detects JavaScript-only rendering, and produces a shareable report card. Under “Going further” it also lists the newer, optional signals it doesn't score: llms.txt, agents.json, a markdown copy for AI agents, and forms an agent can fill in. No signup needed for the basic check.

## What to fix first

If your score is low, prioritize in this order:

1. **Unblock the crawlers that feed AI answers** in robots.txt and at the CDN — no other fix matters if AI can't reach you at all.
2. **Fix JS-only rendering** if you have it — this makes your content visible for the first time.
3. **Add structured data** — takes 5 minutes with our Schema Wizard, huge upside for how you're cited.
4. **Add meta descriptions and OG tags** — table stakes for both SEO and AI.
5. **Optional: add llms.txt** — cheap to add, but few AI crawlers request it yet, so the check lists it without scoring it.

Everything above is free. The paid tier of Crawl Readiness ($29/mo Pro) adds LLM answer monitoring — tracking whether ChatGPT, Claude, Perplexity and Google AI mention your brand for the questions your customers ask, with a trend line. That's the ROI story you can bring to your boss.

### Try Crawl Readiness free

Check whether ChatGPT, Claude, Perplexity, and 50+ other AI crawlers can access your website — no signup for the basics.

[Run a free readiness check →](https://www.crawlreadiness.com/)

---

Source: https://www.crawlreadiness.com/blog/how-to-check-ai-crawlers