Perplexity's AI crawlers
Perplexity documents each of these and says what it is for, so a site can say yes to one and no to another. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
PerplexityBot | AI search index | Counted | 14.8% | 8.4% |
Perplexity-User | Fetches when a user asks | Counted | 12.7% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
PerplexityBot · AI search index
Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Perplexity-User · Fetches when a user asks
Visits a page when a user's question needs it and links to that page in the answer. Perplexity says it generally ignores robots.txt because a user asked for the fetch.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Which of Perplexity's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.