OAI-SearchBot
Surfaces websites in ChatGPT's search features. OpenAI says sites that disallow it will not be shown in ChatGPT search answers.
Where this comes from
- Documented by the vendor. Source: developers.openai.com
- Honours robots.txt. Published addresses: openai.com/searchbot.json.
- Counted in the readiness score. It can keep a site out of AI search or answers, so a block costs points.
In the top 1,000 websites
- 10.2% block it in robots.txt (58 of the 569 sites with a robots.txt), as of 2026-10-12.
- 9.8% refuse it at the server, answering its user agent with a 401, 403, 429 or 503 while a browser gets the page.
- 8% allow it in robots.txt and refuse it anyway (44 sites): a firewall or CDN rule the site owner may not know about. The reverse also happens: 34 sites block it in robots.txt and still serve it.
- Method, limits and the full data: the October 2026 study of the top 1,000 websites.
What blocking it costs
Your pages leave the search index OAI-SearchBot builds, so OpenAI's product can no longer find or cite them from its own index. The score counts it among the crawlers that feed AI search and answers, so a block costs points.
The robots.txt lines
To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):
User-agent: OAI-SearchBot
Allow: /To keep it out:
User-agent: OAI-SearchBot
Disallow: /A rule in robots.txt is a request. OpenAI says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.
Also from OpenAI
GPTBot· Training onlyChatGPT-User· Fetches when a user asks
Does OAI-SearchBot get through on your site?
Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.