Line-by-line robots.txt audit

Paste a URL or your robots.txt and we'll find syntax errors, orphaned rules, empty user-agent blocks, and unknown directives — plus a summary of which AI crawlers are explicitly listed vs. relying on your wildcard. Free with a Crawl Readiness account.

What we check

Syntax — every line matches Directive: value
Known directives — flags unknown ones (typos, custom rules crawlers ignore)
Empty user-agent groups — User-agent declared but no rules follow before a blank line closes the group
Orphaned rules — Allow/Disallow lines that appear before any User-agent
Path formatting — Allow/Disallow values that don't start with /
Crawl-delay values — must be numeric
Noindex directive — flags the common mistake of putting Noindex in robots.txt (Google stopped honoring it in 2019)
Wildcard traps — User-agent: * Disallow: / that would block every crawler including Google
AI-crawler coverage — how many of the tracked AI bots you named explicitly vs. rely on the wildcard rule for

Two ways to check

By URL — we fetch yourdomain.com/robots.txt and audit it
Paste text — audit a robots.txt you're editing locally before deploying