Tool · robots.txt Validator

Line-by-line robots.txt audit

Paste a URL or your robots.txt and we'll find syntax errors, orphaned rules, empty user-agent blocks, and unknown directives — plus a summary of which AI crawlers are explicitly listed vs. relying on your wildcard. Free. No signup required.

What we check

  • Syntax — every line matches Directive: value
  • Known directives — flags unknown ones (typos, custom rules crawlers ignore)
  • Empty user-agent groups — User-agent declared but no rules follow before a blank line closes the group
  • Orphaned rules — Allow/Disallow lines that appear before any User-agent
  • Path formatting — Allow/Disallow values that don't start with /
  • Crawl-delay values — must be numeric
  • Noindex directive — flags the common mistake of putting Noindex in robots.txt (Google stopped honoring it in 2019)
  • Wildcard traps — User-agent: * Disallow: / that would block every crawler including Google
  • AI-crawler coverage — how many of the tracked AI bots you named explicitly vs. rely on the wildcard rule for

Two ways to check

  1. By URL — we fetch yourdomain.com/robots.txt and audit it
  2. Paste text — audit a robots.txt you're editing locally before deploying