Control which AI crawlers can read your site

Paste your site URL. We'll fetch your existing robots.txt, show you what's there, and let you set per-crawler rules for 32+ AI bots — allow, block, delay, or use the new Content-Signal directive. Output is a merged robots.txt you can drop straight in.

What is robots.txt?

robots.txt is a plain-text file at the root of your website (like yourdomain.com/robots.txt) that tells crawlers which parts of your site they can access. It's the primary way to control how AI systems like ChatGPT, Claude, Perplexity, and Google's Gemini interact with your content.

Why AI-specific rules matter

Google's regular Googlebot is separate from Google-Extended, which controls Gemini training
OpenAI has three bots: GPTBot (training), OAI-SearchBot (index), ChatGPT-User (live browsing)
Anthropic, Perplexity, xAI, and others all follow the same split
The new Content-Signal directive lets you declare granular preferences (train/search/ai-input) in one line

What we generate

Per-bot Allow / Disallow rules for 32+ AI crawlers
Content-Signal directive for train/search/ai-input preferences
Optional Crawl-delay (rate limit)
Sitemap pointer
Merged with your existing robots.txt — nothing gets deleted