Tool · robots.txt Validator

# Line-by-line robots.txt audit

Paste a URL or your robots.txt and get a per-line audit — syntax errors, empty groups, misconfigured wildcards, unknown directives, and full AI-crawler coverage summary. Free. No signup required.

Paste a URL or your robots.txt and we'll find syntax errors, orphaned rules, empty user-agent blocks, and unknown directives — plus a summary of which AI crawlers are explicitly listed vs. relying on your wildcard. Free. No signup required.

## What we check

- **Syntax** — every line matches `Directive: value`
- **Known directives** — flags unknown ones (typos, custom rules crawlers ignore)
- **Empty user-agent groups** — `User-agent` declared but no rules follow before a blank line closes the group
- **Orphaned rules** — `Allow`/`Disallow` lines that appear before any `User-agent`
- **Path formatting** — `Allow`/`Disallow` values that don't start with `/`
- **Crawl-delay values** — must be numeric
- **Noindex directive** — flags the common mistake of putting Noindex in robots.txt (Google stopped honoring it in 2019)
- **Wildcard traps** — `User-agent: * Disallow: /` that would block every crawler including Google
- **AI-crawler coverage** — how many of the tracked AI bots you named explicitly vs. rely on the wildcard rule for

## Two ways to check

1. **By URL** — we fetch `yourdomain.com/robots.txt` and audit it
2. **Paste text** — audit a robots.txt you're editing locally before deploying

---

Source: https://www.crawlreadiness.com/tools/robots-validator