Cloudflare Blocking AI Crawlers: How to Fix It
If your site is on Cloudflare and never appears in ChatGPT Search or Perplexity answers, a bot rule may be returning 403 or a challenge page to AI crawlers. Test it first:
Free: free single-URL checks, up to 10 per hour, no sign-up.
Short answer: Cloudflare can block AI crawlers through its “Block AI bots” / AI Crawl Control setting, Bot Fight Mode, Super Bot Fight Mode or custom WAF rules. To stay visible in AI search, allow search and retrieval bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot) and decide separately about training bots (GPTBot, CCBot, Google-Extended).
Step 1 — Confirm the block
Run the AI crawler checker on an important URL. Look for a browser status of 200 next to a bot status of 403, 429 or 503, or a page titled “Just a moment…”. That pattern is a WAF block, not a robots.txt issue. You can also test from a terminal:
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://example.com/Step 2 — Review Cloudflare bot settings
- Open your domain in the Cloudflare dashboard.
- Under AI Crawl Control (or Security → Bots), check whether AI crawlers are set to block and adjust per crawler.
- Check whether Bot Fight Mode is on; it challenges many automated clients and has limited per-bot exceptions on the free plan.
- Under Security → WAF → Custom rules, look for rules matching user-agent strings such as “GPTBot”, “bot” or “crawler”.
Step 3 — Allow search bots, choose about training bots
| User-agent | Operator | What it is used for | Typical recommendation for visibility |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT Search results and citations | Allow |
GPTBot | OpenAI | Model training data | Your choice |
ChatGPT-User | OpenAI | Live fetches when a ChatGPT user asks about a URL | Allow |
PerplexityBot | Perplexity | Perplexity search index and citations | Allow |
ClaudeBot | Anthropic | Claude training and retrieval | Your choice |
Google-Extended | Gemini training/grounding opt-out token (not a separate crawler) | Your choice | |
Applebot-Extended | Apple | Apple Intelligence training opt-out token | Your choice |
Bingbot | Microsoft | Bing index, which also feeds Copilot | Allow |
CCBot | Common Crawl | Open web corpus used by many models | Your choice |
Step 4 — Align robots.txt
Your robots.txt should express the same policy. A minimal example is on the AI crawler checker page. Remember that a wildcard Disallow: / under User-agent: * also blocks any AI bot that has no group of its own.
Step 5 — Re-test and monitor
Re-run the check after every change. Settings can be reset by plan changes or new templates, so re-check monthly. Once access is fixed, publish an llms.txt file and complete your structured data. To track whether Gemini and Perplexity actually mention your brand afterwards, see monitoring plans.
FAQ
Does Cloudflare block AI crawlers by default?
Since July 2025 Cloudflare has offered a setting to block AI crawlers that is enabled by default for many new domains, and Bot Fight Mode can also challenge bot traffic. Check Security → Bots and AI Crawl Control in your dashboard.
How do I know Cloudflare is blocking OAI-SearchBot?
Request the page with the OAI-SearchBot user-agent. If a browser gets 200 but the bot gets 403, 503 or a “Just a moment…” challenge page, Cloudflare is blocking it. GEO Checker runs this comparison for 9 bots for free.
Will allowing AI crawlers let them train on my content?
Not necessarily. You can allow search/retrieval bots (OAI-SearchBot, PerplexityBot, ChatGPT-User) and keep training bots (GPTBot, CCBot, Google-Extended) disallowed.
Do I need to change robots.txt too?
Yes — both layers must agree. robots.txt states your policy; Cloudflare enforces access. A bot allowed in robots.txt but blocked by a WAF is still invisible.
Related tools and guides
Cloudflare dashboard names change over time; menu labels are as of October 2026. Not affiliated with Cloudflare.