Tool · robots.txt Generator

# Control which AI crawlers can read your site

Generate an AI-crawler-aware robots.txt in seconds. Pick a preset (allow all, search-only, block training) or customize per-bot. Free with a Crawl Readiness account.

Paste your site URL. We'll fetch your existing `robots.txt`, show you what's there, and let you set per-crawler rules for 51 AI bots — allow, block, delay, or use the new `Content-Signal` directive. Output is a merged robots.txt you can drop straight in.

## What is robots.txt?

**robots.txt** is a plain-text file at the root of your website (like `yourdomain.com/robots.txt`) that tells crawlers which parts of your site they can access. It's the primary way to control how AI systems like ChatGPT, Claude, Perplexity, and Google's Gemini interact with your content.

## Why AI-specific rules matter

- Google's regular `Googlebot` is separate from `Google-Extended`, which controls Gemini training and grounding in the Gemini apps (not Google Search or AI Overviews)
- OpenAI has three bots: `GPTBot` (training), `OAI-SearchBot` (index), `ChatGPT-User` (live browsing)
- Anthropic, Perplexity, xAI, and others all follow the same split
- The new `Content-Signal` directive lets you declare granular preferences (train/search/ai-input) in one line

## What we generate

- Per-bot Allow / Disallow rules for 51 AI crawlers
- Content-Signal directive for train/search/ai-input preferences
- Optional Crawl-delay (rate limit)
- Sitemap pointer
- Merged into your existing robots.txt — rules for other bots are kept; rules for the AI crawlers you set here are replaced so they can't conflict

---

Source: https://www.crawlreadiness.com/tools/robots-txt