AI crawlers · OpenAI · Training only · Not scored

GPTBot

Crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it says a site's content should not be used in training; it has no effect on ChatGPT search or on ChatGPT fetching a page a user asks about.

Where this comes from

  • Documented by the vendor. Source: developers.openai.com
  • Honours robots.txt. Published addresses: openai.com/gptbot.json.
  • Listed, not scored. It cannot keep a site out of today's answers, so the check shows a block without deducting for it.

In the top 1,000 websites

  • 18.5% block it in robots.txt (105 of the 569 sites with a robots.txt), as of 2026-10-12.
  • 14.4% refuse it at the server, answering its user agent with a 401, 403, 429 or 503 while a browser gets the page.
  • 11.1% allow it in robots.txt and refuse it anyway (61 sites): a firewall or CDN rule the site owner may not know about. The reverse also happens: 53 sites block it in robots.txt and still serve it.
  • Method, limits and the full data: the October 2026 study of the top 1,000 websites.

What blocking it costs

Nothing in today's answers, and nothing in the readiness score: GPTBot only gathers training data, so blocking it is a policy choice about future models, not about being found. The check lists the block without deducting for it.

The robots.txt lines

To let it in (a group that names the crawler replaces any wildcard rules for it, so this also lifts a User-agent: * block):

User-agent: GPTBot
Allow: /

To keep it out:

User-agent: GPTBot
Disallow: /

A rule in robots.txt is a request. OpenAI says this crawler honours it. The robots.txt generator writes a whole file with the same split: block the training crawlers, keep the search and answer crawlers.

Go deeper

Are you accidentally blocking GPTBot? How to check and fix it

Also from OpenAI

Does GPTBot get through on your site?

Run the free check: it reads your robots.txt against all 51 crawlers and tells you which ones can read you, and what to change.