Track AI bot traffic

See when AI crawlers and assistants — GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User, OAI-SearchBot, Bytespider, CCBot, and more — fetch your pages, and tell apart an AI assistant live-fetching a page for a user’s question from an AI crawler indexing you for training.

⚠️

AI bots do not run JavaScript, so the browser snippet can never see them. The only way to record an AI-bot hit is to capture the request server-side at your origin and forward it to TrackCrumb. This page shows how.

How it works

A request arrives at your server with a bot User-Agent (e.g. ChatGPT-User/1.0).
Your server does a coarse check — does the UA look like a bot? — and if so, fires a fire-and-forget event to the TrackCrumb ingest endpoint (POST /e) with the raw user_agent.
The ingest service does the authoritative classification: it matches the UA against the maintained AI-bot allowlist, stamps $is_bot / $bot_name / $bot_category, and records the event under the reserved name $bot_pageview — so bot hits stay out of your human funnels, retention, and trends.
The hits show up in the dashboard under Acquisition → AI bot activity, split by bot and by page.

You keep a loose pre-filter on your side and never need to maintain a bot list — the list lives in one place, server-side.

Next.js (Edge middleware)

Create middleware.ts in your project root:

import { NextResponse } from "next/server";
import type { NextRequest } from "next/server";
 
// Coarse pre-filter only — TrackCrumb does the authoritative classification.
const AI_BOT_RE =
  /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-Web|anthropic-ai|PerplexityBot|Perplexity-User|CCBot|Bytespider|Amazonbot|Applebot-Extended|cohere-ai|Meta-ExternalAgent|Meta-ExternalFetcher|Google-CloudVertexBot|DuckAssistBot|MistralAI-User|Diffbot|Timpibot|Omgilibot|YouBot/i;
 
const INGEST = "https://ingest.trackcrumb.com/e";
 
export function middleware(req: NextRequest) {
  const ua = req.headers.get("user-agent") ?? "";
  if (AI_BOT_RE.test(ua)) {
    // Fire-and-forget — never block the response to the bot.
    fetch(INGEST, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "X-API-Key": process.env.TRACKCRUMB_API_KEY ?? "",
      },
      body: JSON.stringify({
        batch: [
          {
            distinct_id: "ai-bot",
            event_name: "$pageview",
            url: req.nextUrl.href,
            referrer: req.headers.get("referer") ?? "",
            user_agent: ua,
            properties: { $server_captured: "true" },
          },
        ],
      }),
    }).catch(() => {});
  }
  return NextResponse.next();
}
 
// Skip static assets — bots only request real pages.
export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

Set TRACKCRUMB_API_KEY (your st_live_... key from Settings → API Keys) in your environment. Done — AI-bot hits start flowing.

distinct_id is a constant here on purpose: bot hits are counted by volume, not by unique visitor, so a single id keeps cardinality flat. The precise bot name is filled in server-side as $bot_name.

Any backend (raw HTTP)

The capture is just a POST to /e. Detect a bot UA in your request handler / reverse proxy / edge worker and send:

curl -X POST https://ingest.trackcrumb.com/e \
  -H "Content-Type: application/json" \
  -H "X-API-Key: st_live_REPLACE_ME" \
  -d '{"batch":[{"distinct_id":"ai-bot","event_name":"$pageview","url":"https://yoursite.com/pricing","user_agent":"Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)","properties":{"$server_captured":"true"}}]}'

Field reference (same contract as the event payload):

Field	Required	Notes
`distinct_id`	yes	Use a constant like `"ai-bot"` — bot hits are volume-counted.
`event_name`	yes	Send `$pageview`; ingest rewrites it to `$bot_pageview` on a bot match.
`user_agent`	yes	The bot’s UA — this is what gets classified. If omitted, the ingest falls back to the request’s `User-Agent` header.
`url`	recommended	The page the bot fetched — powers Top pages accessed by AI.
`referrer`	optional	`Referer` header if present.
`properties`	optional	Free-form; `$server_captured` is just a hint for your own debugging.

Only fire for requests whose UA matches your pre-filter — otherwise you’d double-count human pageviews the browser snippet already captures.

What you’ll see

A new AI bot activity panel on the Acquisition page listing each bot, its type (AI assistant (live fetch) vs AI crawler (training/index)), hit count, and last-seen date.
A Top pages accessed by AI table — answers “did AI fetch this page?”.
Ad-hoc segmentation everywhere: filter any report by $bot_name or $bot_category, or exclude bots entirely (they’re already a separate $bot_pageview event, so your normal reports never include them).

Coverage & limits

Only bots that (a) actually reach your origin and (b) match the allowlist are recorded. New assistants are added to the server-side list over time.
Google-Extended is a robots.txt token, not a User-Agent (Gemini fetches as Googlebot), so it can’t be detected from the UA.
Bots that ignore your server (pure third-party indexes) never hit your origin and so can’t be seen here.

WordPress

The WordPress plugin (0.2.0+) has built-in server-side bot capture — enable Track AI bots under Settings → TrackCrumb → Capture options (on by default). No code required.

Node / Express / Next.js

Use @trackcrumb/sdk-node instead of hand-rolling the fetch:

import { TrackCrumbServer, trackBots, trackBotsEdge } from "@trackcrumb/sdk-node";
 
const client = new TrackCrumbServer({ apiKey: process.env.TRACKCRUMB_API_KEY! });
 
// Express:
app.use(trackBots(client));
 
// Next.js middleware.ts:
export function middleware(req) {
  trackBotsEdge(client, req);
  return NextResponse.next();
}

It ships the same allowlist (classifyBot / isAiBot are exported) and posts $bot_pageview for you.

WordPress plugin Install with AI