OpenAI's AI crawlers
OpenAI documents each of these and says what it is for, so a site can say yes to one and no to another. Numbers are from the October 2026 study of the top 1,000 websites (2026-10-12).
The split
| Token | Job | Score | Blocked by | Allowed but refused at the server |
|---|---|---|---|---|
GPTBot | Training only | Not scored | 18.5% | 11.1% |
OAI-SearchBot | AI search index | Counted | 10.2% | 8% |
ChatGPT-User | Fetches when a user asks | Counted | 12.1% | not tested |
“Blocked by” is the share of the 569 top-1,000 sites with a robots.txt. The last column was measured for four crawlers only.
GPTBot · Training only
Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it says a site's content should not be used in training; it has no effect on ChatGPT search or on ChatGPT fetching a page a user asks about.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
OAI-SearchBot · AI search index
Surfaces websites in ChatGPT's search features. OpenAI says sites that disallow it will not be shown in ChatGPT search answers.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
ChatGPT-User · Fetches when a user asks
Visits a web page when a ChatGPT user or a Custom GPT asks about it; not used for crawling the web automatically. Because the fetch is user-initiated, OpenAI says robots.txt rules may not apply.
Documented by the vendor. Its page has the robots.txt lines and what blocking costs.
Opt out of OpenAI's training, stay in its answers
The lines below block only the crawler that gather training data. OAI-SearchBot, ChatGPT-User stay open, so the site is still found and cited. The readiness score does not change.
User-agent: GPTBot
Disallow: /Which of OpenAI's crawlers can read your site?
Run the free check, or write the whole policy with the robots.txt generator.