AI Crawler Checker
Check in about 20 seconds whether GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and 5 other AI crawlers can reach your page — both by robots.txt rules and by a live request.
Free: free single-URL checks, up to 10 per hour, no sign-up.
An AI crawler checker answers one question: can the bots that feed ChatGPT, Perplexity, Claude, Gemini and Copilot actually fetch your page? If they cannot, those engines cannot quote or cite it, no matter how good the content is.
How to check AI crawler access in 4 steps
- Enter a key URL — a product, pricing or landing page, not just the homepage.
- Read the robots.txt verdicts — we evaluate the rule group that applies to each of the 9 user-agents below, including
User-agent: *fallbacks. - Compare live responses — we request the page as each bot and as a browser. A 200 for the browser but 403/429/503 or a “Just a moment…” page for a bot means your CDN or WAF is blocking it.
- Apply the fix and re-check — edit robots.txt or your firewall rule, then run the check again.
The 9 AI and search crawlers we test
| User-agent | Operator | What it is used for | Typical recommendation for visibility |
|---|---|---|---|
OAI-SearchBot | OpenAI | ChatGPT Search results and citations | Allow |
GPTBot | OpenAI | Model training data | Your choice |
ChatGPT-User | OpenAI | Live fetches when a ChatGPT user asks about a URL | Allow |
PerplexityBot | Perplexity | Perplexity search index and citations | Allow |
ClaudeBot | Anthropic | Claude training and retrieval | Your choice |
Google-Extended | Gemini training/grounding opt-out token (not a separate crawler) | Your choice | |
Applebot-Extended | Apple | Apple Intelligence training opt-out token | Your choice |
Bingbot | Microsoft | Bing index, which also feeds Copilot | Allow |
CCBot | Common Crawl | Open web corpus used by many models | Your choice |
Example robots.txt that stays visible in AI search but opts out of training:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /Read the full walkthrough for fixing Cloudflare rules that block AI crawlers.
AI crawler checker vs. brand mention trackers
| AI crawler checker (this page) | AI mention tracker | |
|---|---|---|
| Question answered | Can AI bots fetch my page? | Do AI answers mention my brand? |
| When to use | First — it is a prerequisite | After access is fixed, to track results |
| GEO Checker | Free | Weekly on Gemini + Perplexity, from $19/month (see pricing) |
FAQ
What is an AI crawler checker?
An AI crawler checker tests whether bots such as OAI-SearchBot, GPTBot, PerplexityBot and ClaudeBot are allowed by a site’s robots.txt and can actually download the page without being blocked by a firewall or CDN.
How do I check if GPTBot is blocked on my site?
Enter your URL above. GEO Checker reads your robots.txt rules for GPTBot and 8 other agents, then requests the page with the GPTBot user-agent and compares the HTTP status and content with a normal browser request. A 403, 429 or challenge page means a WAF is blocking it.
Should I block GPTBot but allow OAI-SearchBot?
Many publishers do. GPTBot collects training data; OAI-SearchBot powers ChatGPT Search results and citations. Blocking GPTBot does not remove you from ChatGPT Search as long as OAI-SearchBot is allowed.
Why does robots.txt say “allowed” but the bot is still blocked?
robots.txt is only a request. Cloudflare Bot Fight Mode, “Block AI bots” settings, Shopify or Wix firewall rules and rate limits can return 403 or a JavaScript challenge to AI user-agents regardless of robots.txt.
Is the AI crawler check free?
Yes: free single-URL checks, up to 10 per hour, no sign-up.