Should You Block GPTBot in robots.txt?

Updated 2026-10-24 · By the GEO Checker team

Short answer: blocking GPTBot stops OpenAI from using your pages to train future models, but it does not remove you from ChatGPT Search as long as you allow OAI-SearchBot. Whether to block it depends on how much you value training exposure versus control over your content. This guide explains the trade-offs and gives ready-to-use robots.txt examples.

What GPTBot is (and what it is not)

GPTBot is OpenAI’s web crawler for collecting publicly available content that may be used to improve and train its generative AI models. Its full user-agent string contains GPTBot, and OpenAI publishes the IP ranges it uses so you can verify genuine requests. GPTBot respects robots.txt.

It is easy to confuse GPTBot with OpenAI’s other agents:

Many sites blocked “ChatGPT” in 2023 by disallowing GPTBot, which was the only OpenAI crawler at the time. Since OpenAI introduced OAI-SearchBot, those rules no longer affect search visibility on their own — but over-broad rules such as AI-blocker plugin lists often block all three agents together. Check yours with the AI crawler checker.

Step by step

1. Check what your robots.txt does today. Run the AI crawler checker or the full AI visibility check on an important page. It reports, bot by bot, whether GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others are allowed, and which rule matched.
2. Decide what you want from AI training. Ask whether you benefit from models “knowing” your brand and content without a live search. Publishers selling original content, paywalled sites and sites with licensing deals often block training. Software, e-commerce and service businesses usually gain from broad awareness and may allow it.
3. Keep search and user-fetch bots allowed. Whatever you decide about GPTBot, allow OAI-SearchBot and ChatGPT-User if you want to be cited in ChatGPT answers. Do the same for PerplexityBot, Bingbot and other search crawlers you care about.
4. Write explicit groups for each bot. Give each crawler its own User-agent group. Crawlers obey the most specific group that matches them, so a GPTBot group overrides “User-agent: *”. Avoid relying on wildcard groups to block AI bots — it is easy to block search crawlers by accident.
5. Check firewall rules match your robots.txt. If you use Cloudflare or another CDN, make sure its AI bot settings follow the same decision. Cloudflare can block AI crawlers at the edge regardless of robots.txt. See fixing Cloudflare blocking AI crawlers.
6. Verify real crawler behavior after the change. Use AI crawler analytics or your server logs to confirm GPTBot visits stop (if blocked) while OAI-SearchBot continues to fetch pages. Verify requests against OpenAI’s published IP ranges, because scrapers often fake the GPTBot user-agent.

robots.txt examples for each choice

1. Allow everything (maximum AI exposure)

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

2. Block training, keep ChatGPT Search (most common business choice)

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: *
Allow: /

3. Allow training only for public marketing pages

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Disallow: /

User-agent: *
Allow: /

4. Block all OpenAI crawlers (not recommended if you want AI traffic)

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

Pros and cons of blocking GPTBot

Two facts are worth stressing. First, blocking only applies going forward; it does not delete what was already collected. Second, robots.txt is a voluntary standard. Reputable crawlers like GPTBot follow it, but scrapers that fake user-agents do not, which is why verifying IP addresses matters.

A note on other training crawlers

The same decision applies to other AI companies. ClaudeBot (Anthropic), Google-Extended (a robots.txt token controlling Gemini training and grounding use, not a separate crawler), Applebot-Extended (Apple Intelligence training) and CCBot (Common Crawl, used by many model builders) can each be allowed or disallowed independently. Blocking Google-Extended does not affect Google Search ranking. If you publish an llms.txt file, keep it consistent with your robots.txt decision. For online stores, the AI shopping agent readiness check shows whether shopping-related agents can reach product pages.