Bots & AI crawlers

Numative reports bot and AI crawler traffic (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) separately from human traffic, and marks each visit verified (confirmed by source IP) or claimed (self-identified user-agent only). There are two layers, because a JavaScript tracker cannot see most AI crawlers: they fetch HTML and never execute scripts.

Automatic classification

Every event that reaches Numative is classified by user-agent into a bot kind: AI crawlers, search engine bots, link-preview bots and SEO crawlers. Bots that do execute JavaScript (headless browsers, some preview bots) therefore appear in the Bots & AI crawlers breakdown with no setup at all.

• Bots are excluded from your human metrics by default. Turn on Include bot traffic in site settings to count them, or filter any view by the Bots & AI crawlers dimension.
• Uptime monitors and generic HTTP tools are dropped before storage. They have no analytical value and would burn your event quota.
• The Collection source dimension separates visits seen by the JS tracker from visits captured server-side.

Capturing AI crawlers that never run JavaScript

AI crawlers request your HTML server-side, so their visits must be observed server-side too. Numative accepts a server feed that stores bot traffic only: human requests are ignored by the feed (humans stay on the JS tracker), so nothing is double-counted. Server-captured visits count toward your plan's event quota.

1. 1

   Generate an ingest key

   Open your site's Settings tab, find AI crawler tracking and click Generate ingest key. The key is shown once; the in-app card fills it into ready-to-paste snippets and the log-drain URL. A write-scoped API key works too.

2. 2

   Connect one feed

   Pick the option below that matches your stack. One feed per site is enough. Once visits arrive, the setup card shows a live counter and the Bots & AI crawlers report fills in.

WordPress

Install the Numative Analytics plugin and paste your Site ID and ingest key. It forwards crawler visits server-side without blocking the response and excludes bot user-agents from caching automatically (WP Rocket and LiteSpeed; for W3 Total Cache and WP Super Cache it shows the exact list to paste). No log access required, works on shared hosting.

Without the plugin (or on non-WordPress PHP), drop this into functions.php or an mu-plugin:

<?php // functions.php or an mu-plugin (or just install the Numative plugin)
add_action('template_redirect', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match(/gptbot|oai-searchbot|chatgpt-user|claudebot|anthropic-ai|perplexitybot|google-extended|ccbot|bytespider|amazonbot|meta-externalagent/i, $ua)) return;
  if (!defined('DONOTCACHEPAGE')) define('DONOTCACHEPAGE', true); // don't cache bots
  wp_remote_post('https://app.numative.com/api/v1/collect', [
    'blocking' => false, 'timeout' => 0.5,
    'headers' => ['Authorization' => 'Bearer YOUR_INGEST_KEY', 'Content-Type' => 'application/json'],
    'body' => wp_json_encode(['site_id' => 'YOUR_SITE_ID', 'events' => [[
      'path' => $_SERVER['REQUEST_URI'], 'userAgent' => $ua,
      'ip' => $_SERVER['REMOTE_ADDR'],
    ]]]),
  ]);
}, 1);

Cloudflare

Behind Cloudflare's edge cache, crawler requests may never reach your origin. A Worker runs on every request, cache hits included, and works on the free plan. Deploy this and route it to your zone:

// Cloudflare Worker - runs on every request, even edge-cache hits.
export default {
  async fetch(request, env, ctx) {
    const res = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    if (/gptbot|oai-searchbot|chatgpt-user|claudebot|anthropic-ai|perplexitybot|google-extended|ccbot|bytespider|amazonbot|meta-externalagent/i.test(ua)) {
      const url = new URL(request.url);
      ctx.waitUntil(fetch("https://app.numative.com/api/v1/collect", {
        method: "POST",
        headers: { "content-type": "application/json", authorization: "Bearer YOUR_INGEST_KEY" },
        body: JSON.stringify({ site_id: "YOUR_SITE_ID", events: [{
          path: url.pathname, userAgent: ua, ip: request.headers.get("cf-connecting-ip"),
        }] }),
      }));
    }
    return res;
  },
};

Cloudflare Logpush to the log-drain URL below also works, but Logpush is Enterprise-only.

Vercel

No code. Send the request log to Numative:

1. Vercel → your project → Settings → Log Drains → Add.
2. Delivery format: NDJSON. Sources: Requests.
3. Paste your ingest URL as the endpoint and save:

https://app.numative.com/api/ingest/logs?site=YOUR_SITE_ID&key=YOUR_INGEST_KEY

Numative parses each line, keeps only real bot and crawler requests, verifies them by IP, and drops everything else.

Node and other backends

Forward requests with an AI-bot user-agent to POST /api/v1/collect, authenticated with the ingest key. For Next.js, middleware is the natural place:

// middleware.ts
import { NextResponse, type NextRequest } from "next/server";
const AI_UA = /gptbot|oai-searchbot|chatgpt-user|claudebot|anthropic-ai|perplexitybot|google-extended|ccbot|bytespider|amazonbot|meta-externalagent/i;
export function middleware(req: NextRequest) {
  const ua = req.headers.get("user-agent") ?? "";
  if (AI_UA.test(ua)) {
    void fetch("https://app.numative.com/api/v1/collect", {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer YOUR_INGEST_KEY` },
      body: JSON.stringify({ site_id: "YOUR_SITE_ID", events: [{
        path: req.nextUrl.pathname, userAgent: ua,
        ip: req.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      }] }),
    });
  }
  return NextResponse.next();
}

The endpoint accepts up to 500 events per request with path, userAgent and optional ip, referrer, hostname and timestamp per event. See the Stats API reference for the full schema.

Verified vs claimed

Anyone can send a fake GPTBot user-agent, so the report distinguishes:

A spoofed user-agent is harmless: it is stored as claimed, never verified, and stays out of your human metrics either way.

Which crawlers the snippets match

The snippets above forward: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, anthropic-ai (Anthropic); PerplexityBot; Google-Extended; CCBot (Common Crawl); Bytespider (ByteDance); Amazonbot; and Meta-ExternalAgent. Numative classifies whatever reaches it, so you can widen the regex if you want search engine bots from your origin too; the log-drain path already sees everything.