How AI reads your site, without the hype
Guides, deep dives, and practical tips on making your website accessible to ChatGPT, Claude, Perplexity, and every AI crawler that matters.
AI crawler access across the top 1,000 websites (October 2026)
One pass over the Tranco top 1,000: which AI crawlers robots.txt blocks, how many servers turn AI crawlers away even when robots.txt allows them, and how far llms.txt, markdown copies and structured data have spread.
How to block AI training without disappearing from AI answers
Every major AI company runs one crawler for training and others for search and answers. Block the nine training-only crawlers, leave the rest open, and you are out of the training sets but still in the answers. Here is the list and the robots.txt.
Why ChatGPT and Perplexity disagree about your brand (and what moves each one)
Ask ChatGPT and Perplexity the same question and one names you while the other doesn't. That's not a contradiction: one answers from memory, the other from a search it just ran. Here's what each is made of, what moves it, and how to read the difference.
GEO vs SEO: what's actually different (and what stays the same)
Generative engine optimization (GEO) and SEO overlap more than you'd think — but the win condition is different. Here's how they compare and what to do about it.
Does ChatGPT read my website? How to check in 60 seconds
ChatGPT can only cite pages it can actually fetch. Here's how ChatGPT accesses websites, how to check whether yours is readable, and the three things that usually block it.
Are you blocking GPTBot on purpose? How to check, and how to choose
GPTBot is OpenAI's training crawler. Blocking it keeps your pages out of future model training and does not take you out of ChatGPT search. Here's how to tell whether you block it, whether you should, and how to make it a decision rather than an accident.
What is llms.txt? The complete 2026 guide
llms.txt is a proposed standard for helping AI systems understand your website. Here's what it is, why it matters, and how to add one in under 10 minutes.
How to check if ChatGPT and other AI crawlers can access your website
A step-by-step guide to checking whether ChatGPT, Claude, Perplexity and Google AI can access your site, plus the five most common blockers and how to fix each one.
robots.txt for AI in 2026: block, allow, or something in between
Every AI provider has multiple user-agents doing different jobs. Here's the definitive robots.txt guide for ChatGPT, Claude, Google AI, Perplexity, and 40+ others.