PerplexityBot
Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
Where this comes from
- Documented by the vendor. Source: docs.perplexity.ai
- Honours robots.txt. Published addresses: www.perplexity.com/perplexitybot.json.
- Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.
In the top 1,000 websites
- 14.8% block it in robots.txt (84 of the 569 sites with a robots.txt), as of 2026-10-12.
- 11.3% refuse it at the server, answering its user agent with a 401, 403, 429 or 503 while a browser gets the page.
- 8.4% allow it in robots.txt and refuse it anyway (46 sites): a firewall or CDN rule the site owner may not know about. The reverse also happens: 41 sites block it in robots.txt and still serve it.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
Your pages leave the search index PerplexityBot builds, so Perplexity's product can no longer find or cite them from its own index. The score counts it among the crawlers that feed AI search and answers, so a block costs points.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: PerplexityBot
Allow: /To keep it out:
User-agent: PerplexityBot
Disallow: /A rule in robots.txt is a request. Perplexity says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Also from Perplexity
Perplexity-User· Fetches when a user asks
Does PerplexityBot get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.