How to Check if AI Bots Crawl Your Site

Updated 2026-10-24 · By the GEO Checker team

There are two different questions hiding in “do AI bots crawl my site?”: can they (are they allowed and not blocked?) and do they (are real, verified AI crawlers actually requesting your pages?). You need both answers. This guide shows how to test access in seconds, how to read your logs for real visits, and how to tell genuine AI crawlers from scrapers using a fake user-agent.

The AI crawlers worth knowing

Note that Google-Extended and Applebot-Extended are robots.txt tokens, not crawlers: they never appear in your logs. Google’s AI features use the normal Googlebot crawl, so Googlebot visits are what you will see.

Step by step

1. Test whether AI bots can access your pages. Run the AI crawler checker on your homepage and a key page. It reads robots.txt for nine AI bots and fetches the page with each bot’s user-agent, comparing the response with a normal browser to catch firewall blocks, challenges and empty responses.
2. Check your robots.txt manually. Open yourdomain.com/robots.txt and look for groups naming AI user-agents or a “User-agent: *” group with Disallow: /. Remember the most specific matching group wins.
3. Look for AI user-agents in your server logs. Download your access logs from your host or CDN and search for the user-agent names. On a Linux server, a command like grep -Ei "gptbot|oai-searchbot|chatgpt-user|claudebot|perplexitybot|ccbot" access.log lists matching requests.
4. Count visits by bot and status code. Group the matches by bot and HTTP status. Many 200 responses mean bots read your pages; 403, 429 or 503 responses mean something is blocking or throttling them.
5. Verify the IP addresses. Anyone can fake a user-agent. Compare request IPs with the ranges each company publishes (OpenAI, Perplexity and others publish JSON lists) or use reverse DNS for Bingbot and Googlebot. Only verified requests count as real AI crawls.
6. Find important pages AI bots never visit. Compare crawled URLs with your sitemap. Pages that are never fetched may need better internal links, a sitemap entry or a mention in llms.txt.
7. Monitor continuously. Logs are tedious to check by hand. AI crawler analytics records AI bot visits server-side, verifies IPs, flags spoofed traffic and lists uncrawled pages — free for one site.

Example: reading a log line

20.171.207.130 - - [12/Oct/2026:08:14:03 +0000] "GET /pricing/ HTTP/1.1" 200 18234 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot"

This line says a client identifying as GPTBot requested /pricing/ and received a 200 response. To trust it, check that the IP address falls inside OpenAI’s published GPTBot ranges. If the same user-agent comes from a residential ISP or an unrelated cloud provider, it is almost certainly a scraper.

Example: counting AI bot hits by status

grep -Ei "gptbot|oai-searchbot|claudebot|perplexitybot" access.log \
  | awk '{print $9}' | sort | uniq -c

Output such as 412 200 and 96 403 means roughly one in five AI bot requests is being refused. That usually points to a bot-protection or WAF rule; see fixing Cloudflare blocking AI crawlers.

Reading the results

Server-side data is far more reliable than JavaScript analytics for this job: AI crawlers usually do not run JavaScript, so tools like Google Analytics never see them. For stores, combine crawl data with the AI shopping agent readiness check to make sure product pages and feeds are reachable too.