AI Crawler Analytics

Google Analytics for AI bots. See exactly when GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Googlebot visit your site, which pages they read, and whether they were real or spoofed.

Free: 1 site, 7 days of history. Monthly plans: 90 days and more sites.

Why your analytics cannot see AI bots

Tools like Google Analytics, Plausible or Matomo work by running a JavaScript tag in the visitor’s browser. AI crawlers do not behave like browsers. GPTBot, ClaudeBot or PerplexityBot request the HTML of a page, maybe your robots.txt and sitemap, and leave. They never execute the tracking script, so they never appear in your reports.

The only place these visits are visible is on the server or at the edge, where every request arrives regardless of JavaScript. Many hosts do not expose raw logs in a usable form. That is the gap AI Crawler Analytics fills.

How it works

1. Add a tiny snippet

Sign in, add your domain and copy a snippet for a Cloudflare Worker, Next.js middleware, Express or a WordPress must-use plugin. It runs on your server, checks the user-agent and forwards only likely bot requests.

2. We identify the bot

Each hit is matched against a list of 25 AI and search crawlers, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, Applebot-Extended, CCBot, Bytespider, Amazonbot, Meta-ExternalAgent, DuckAssistBot, cohere-ai and MistralAI-User. Human traffic is discarded.

3. We verify the IP

User-agents are easy to fake. We check the request IP against the official ranges published by OpenAI, Google, Microsoft, Perplexity, Apple and DuckDuckGo, and label each hit verified, spoofed or unverified when the operator publishes no list.

4. You see the picture

Your account page shows hits per bot, a daily trend, top pages crawled, 403 and other blocked responses, and the pages in your sitemap that no AI bot has fetched yet.

What you learn from it

Are AI search bots reaching you at all? If OAI-SearchBot and PerplexityBot never show up, ChatGPT Search and Perplexity cannot cite fresh content from your site.
Are you blocking them by accident? A wall of 403 responses for verified bots usually means a WAF or CDN rule. Our Cloudflare guide explains how to fix it, and the AI crawler checker tests a URL instantly.
Which pages do AI engines care about? Top crawled pages show what models read most. Pages that are never crawled are candidates for better internal links, a sitemap entry or an llms.txt mention.
Who is pretending to be a bot? Spoofed counts show how much “bot traffic” is really scrapers using a fake GPTBot or Googlebot user-agent.

Privacy and performance

The snippets send four fields only: user-agent, IP, path and status code. No cookies, no query strings, no visitor identifiers. The IP is used once for verification and is not stored; we keep daily aggregates. Requests are fire-and-forget with a short timeout, so a slow or unreachable collector can never delay your pages. Ordinary visitors are filtered out on your own server before anything is sent.

A note on Google-Extended: it is a robots.txt token that controls whether Google may use your content for Gemini training. Google does not crawl with a separate Google-Extended user-agent, so it never appears in logs; you will see Googlebot instead.

FAQ

What is AI crawler analytics?

AI crawler analytics records every request that AI bots such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot make to your server, then shows which bots came, which pages they fetched, what status code they got and whether the request really came from the bot operator.

Why does Google Analytics not show GPTBot or ClaudeBot?

JavaScript analytics only counts visitors that execute JavaScript and load the tracking tag. AI crawlers download the HTML and leave; they do not run your analytics script, so they are invisible in Google Analytics, Plausible or similar tools. You need server-side or edge logging to see them.

How do you tell a real GPTBot from a fake one?

OpenAI, Google, Microsoft, Perplexity, Apple and DuckDuckGo publish the IP ranges their crawlers use. We match each hit against those official lists (refreshed daily). A hit that claims to be GPTBot but comes from another network is marked spoofed. Bots whose operators publish no list, such as ClaudeBot, are marked unverified.

Does the snippet slow down my site?

No. Every snippet sends the hit after or alongside the response, without awaiting it, with a timeout under two seconds. Only requests whose user-agent looks like a bot are forwarded, so normal visitors cause no extra work.

What data is collected?

Only the user-agent, IP address, path and response status of requests that look like bots. No cookies, no query strings, no visitor data. IP addresses are used for verification at ingest and are not stored; only aggregated counts are kept.

Is it free?

Yes. A free account tracks one site with 7 days of history. Monthly plans keep 90 days and track up to 1 (Starter), 3 (Pro) or 10 (Agency) sites.