[AI crawlers](https://www.crawlreadiness.com/crawlers) · OpenAI · AI search index · Counted in the score

# OAI-SearchBot

OAI-SearchBot is OpenAI's ai search index crawler token. Blocking it costs points in the AI readiness score. 10.2% of the top 1,000 websites with a robots.txt block it.

Surfaces websites in ChatGPT's search features. OpenAI says sites that disallow it will not be shown in ChatGPT search answers.

## Where this comes from

- **Documented by the vendor.** Source: [developers.openai.com](https://developers.openai.com/api/docs/bots)
- **Honours robots.txt.** Published addresses: [openai.com/searchbot.json](https://openai.com/searchbot.json).
- **Counted in the readiness score.** It can keep a site out of AI search or answers, so a block costs points.

## In the top 1,000 websites

- **10.2% block it in robots.txt** (58 of the 569 sites with a robots.txt), as of 2026-10-12.
- **9.8% refuse it at the server**, answering its user agent with a 401, 403, 429 or 503 while a browser gets the page.
- **8% allow it in robots.txt and refuse it anyway** (44 sites): a firewall or CDN rule the site owner may not know about. The reverse also happens: 34 sites block it in robots.txt and still serve it.
- Method, limits and the full data: [the October 2026 study of the top 1,000 websites](https://www.crawlreadiness.com/research/ai-crawler-access-october-2026).

## What blocking it costs

Your pages leave the search index OAI-SearchBot builds, so OpenAI's product can no longer find or cite them from its own index. The score counts it among the crawlers that feed AI search and answers, so a block costs points.

## The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a `User-agent: *` block):

```
User-agent: OAI-SearchBot
Allow: /
```

To keep it out:

```
User-agent: OAI-SearchBot
Disallow: /
```

A rule in robots.txt is a request. OpenAI says this crawler honours it. The [robots.txt generator](https://www.crawlreadiness.com/tools/robots-txt) writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

## Also from [OpenAI](https://www.crawlreadiness.com/crawlers/vendor/openai)

- [`GPTBot`](https://www.crawlreadiness.com/crawlers/gptbot) · Training only
- [`ChatGPT-User`](https://www.crawlreadiness.com/crawlers/chatgpt-user) · Fetches when a user asks

## Does OAI-SearchBot get through on your site?

[Run the free check](https://www.crawlreadiness.com/): it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.

---

Source: https://www.crawlreadiness.com/crawlers/oai-searchbot