Tool · robots.txt Generator

Control which AI crawlers can read your site

Paste your site URL. We'll fetch your existing robots.txt, show you what's there, and let you set per-crawler rules for 51 AI bots — allow, block, delay, or use the new Content-Signal directive. Output is a merged robots.txt you can drop straight in.

What is robots.txt?

robots.txt is a plain-text file at the root of your website (like yourdomain.com/robots.txt) that tells crawlers which parts of your site they can access. It's the primary way to control how AI systems like ChatGPT, Claude, Perplexity, and Google's Gemini interact with your content.

Why AI-specific rules matter

  • Google's regular Googlebot is separate from Google-Extended, which controls Gemini training and grounding in the Gemini apps (not Google Search or AI Overviews)
  • OpenAI has three bots: GPTBot (training), OAI-SearchBot (index), ChatGPT-User (live browsing)
  • Anthropic, Perplexity, xAI, and others all follow the same split
  • The new Content-Signal directive lets you declare granular preferences (train/search/ai-input) in one line

What we generate

  • Per-bot Allow / Disallow rules for 51 AI crawlers
  • Content-Signal directive for train/search/ai-input preferences
  • Optional Crawl-delay (rate limit)
  • Sitemap pointer
  • Merged into your existing robots.txt — rules for other bots are kept; rules for the AI crawlers you set here are replaced so they can't conflict