Bots & AI crawlers

Numative reports bot and AI crawler traffic (GPTBot, ClaudeBot, PerplexityBot, Googlebot and others) separately from human traffic, and marks each visit verified (confirmed by source IP) or claimed (self-identified user-agent only). There are two layers, because a JavaScript tracker cannot see most AI crawlers: they fetch HTML and never execute scripts.

Automatic classification

Every event that reaches Numative is classified into a bot kind: AI crawlers, search engine bots, link-preview bots and SEO crawlers. Bots that do execute JavaScript therefore appear in the Bots & AI crawlers breakdown with no setup at all.

The user agent is only the first signal. A scraper that sends "Chrome" is caught by evidence it cannot rewrite: automation flags a real browser never sets, a GPU-less renderer, a window with no browser chrome above the page, a container with no timezone, and the source address belonging to a datacenter or a known scraper fleet rather than to a consumer connection. Every signal is scored server-side and precision-first, because mislabeling a real visitor as a bot is worse than missing one. Residential ranges, mobile carriers and Apple's Private Relay are deliberately carved out.

• Bots are excluded from your human metrics by default. Turn on Show bot & AI crawler traffic in site settings to see them as their own line on the chart and their own stat, or filter any view by the Bots & AI crawlers dimension. They are never mixed into your visitor counts either way.
• With that setting on, the Overview gains a Traffic by type card: every hit split by who made it, and then the same traffic split by what the crawler came for - answering a reader right now, indexing for later, or harvesting a training corpus that sends nobody. The Pages section gains a Humans/Bots switch, which answers "what are the AI crawlers actually reading" page by page.
• Uptime monitors and generic HTTP tools are dropped before storage. They have no analytical value and would only add noise.
• The Collection source dimension separates visits seen by the JS tracker from visits captured server-side.

Capturing AI crawlers that never run JavaScript

AI crawlers request your HTML server-side, so their visits must be observed server-side too. Numative accepts a server feed that stores bot traffic only: human requests are ignored by the feed (humans stay on the JS tracker), so nothing is double-counted. Server-captured bot visits are never billed.

1. 1

   Generate an ingest key

   Open your site's Settings page, find AI crawler tracking and click Generate ingest key. The key is shown once; the in-app card fills it into ready-to-paste snippets and the log-drain URL. A write-scoped API key works too.

2. 2

   Connect one feed

   Pick the option below that matches your stack. One feed per site is enough. Once visits arrive, the setup card shows a live counter and the Bots & AI crawlers report fills in.

WordPress

Install the Numative Analytics plugin and paste your Site ID and ingest key. It forwards crawler visits server-side without blocking the response and excludes bot user-agents from caching automatically (WP Rocket and LiteSpeed; for W3 Total Cache and WP Super Cache it shows the exact list to paste). No log access required, works on shared hosting.

Without the plugin (or on non-WordPress PHP), drop this into functions.php or an mu-plugin:

<?php // functions.php or an mu-plugin (or just install the Numative plugin)
add_action('template_redirect', function () {
  $ua = $_SERVER['HTTP_USER_AGENT'] ?? '';
  if (!preg_match('/gptbot|oai-searchbot|chatgpt-user|claudebot|claude-web|claude-user|anthropic-ai|perplexitybot|perplexity-user|google-extended|google-cloudvertexbot|bytespider|ccbot|amazonbot|meta-externalagent|meta-externalfetcher|applebot-extended|cohere-ai|cohere-training|diffbot|youbot|imagesiftbot|timpibot|omgili|omgilibot|webzio|webz\.io|firecrawl|img2dataset|ai2bot|kagibot|duckassistbot|brightbot|andibot|aihitbot|semanticscholarbot|velenpublicwebcrawler/i', $ua)) return;
  if (!defined('DONOTCACHEPAGE')) define('DONOTCACHEPAGE', true); // don't cache bots
  wp_remote_post('https://app.numative.com/api/v1/collect', [
    'blocking' => false, 'timeout' => 0.5,
    'headers' => ['Authorization' => 'Bearer YOUR_INGEST_KEY', 'Content-Type' => 'application/json'],
    'body' => wp_json_encode(['site_id' => 'YOUR_SITE_ID', 'events' => [[
      'path' => $_SERVER['REQUEST_URI'], 'userAgent' => $ua,
      'ip' => $_SERVER['REMOTE_ADDR'],
    ]]]),
  ]);
}, 1);

Cloudflare

Behind Cloudflare's edge cache, crawler requests may never reach your origin. A Worker runs on every request, cache hits included, and works on the free plan. Deploy this and route it to your zone:

// Cloudflare Worker - runs on every request, even edge-cache hits.
export default {
  async fetch(request, env, ctx) {
    const res = await fetch(request);
    const ua = request.headers.get("user-agent") || "";
    if (/gptbot|oai-searchbot|chatgpt-user|claudebot|claude-web|claude-user|anthropic-ai|perplexitybot|perplexity-user|google-extended|google-cloudvertexbot|bytespider|ccbot|amazonbot|meta-externalagent|meta-externalfetcher|applebot-extended|cohere-ai|cohere-training|diffbot|youbot|imagesiftbot|timpibot|omgili|omgilibot|webzio|webz\.io|firecrawl|img2dataset|ai2bot|kagibot|duckassistbot|brightbot|andibot|aihitbot|semanticscholarbot|velenpublicwebcrawler/i.test(ua)) {
      const url = new URL(request.url);
      ctx.waitUntil(fetch("https://app.numative.com/api/v1/collect", {
        method: "POST",
        headers: { "content-type": "application/json", authorization: "Bearer YOUR_INGEST_KEY" },
        body: JSON.stringify({ site_id: "YOUR_SITE_ID", events: [{
          path: url.pathname, userAgent: ua, ip: request.headers.get("cf-connecting-ip"),
        }] }),
      }));
    }
    return res;
  },
};

Cloudflare Logpush to the log-drain URL below also works, but Logpush is Enterprise-only.

Vercel

No code. Send the request log to Numative:

1. Vercel → your project → Settings → Log Drains → Add.
2. Delivery format: NDJSON. Sources: Requests.
3. Paste your ingest URL as the endpoint and save:

https://app.numative.com/api/ingest/logs?site=YOUR_SITE_ID&key=YOUR_INGEST_KEY

Numative parses each line, keeps only real bot and crawler requests, verifies them by IP, and drops everything else.

Node and other backends

Forward requests with an AI-bot user-agent to POST /api/v1/collect, authenticated with the ingest key. For Next.js, middleware is the natural place:

// middleware.ts
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
const AI_UA = /gptbot|oai-searchbot|chatgpt-user|claudebot|claude-web|claude-user|anthropic-ai|perplexitybot|perplexity-user|google-extended|google-cloudvertexbot|bytespider|ccbot|amazonbot|meta-externalagent|meta-externalfetcher|applebot-extended|cohere-ai|cohere-training|diffbot|youbot|imagesiftbot|timpibot|omgili|omgilibot|webzio|webz\.io|firecrawl|img2dataset|ai2bot|kagibot|duckassistbot|brightbot|andibot|aihitbot|semanticscholarbot|velenpublicwebcrawler/i;
export function middleware(req: NextRequest, event: NextFetchEvent) {
  const ua = req.headers.get("user-agent") ?? "";
  if (AI_UA.test(ua)) {
    // waitUntil, not a bare fetch: the edge runtime can tear the invocation
    // down as soon as the response is returned, cancelling a request in
    // flight. This never delays the response.
    event.waitUntil(fetch("https://app.numative.com/api/v1/collect", {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer YOUR_INGEST_KEY` },
      body: JSON.stringify({ site_id: "YOUR_SITE_ID", events: [{
        path: req.nextUrl.pathname, userAgent: ua,
        ip: req.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      }] }),
    }).catch(() => {}));
  }
  return NextResponse.next();
}

The endpoint accepts up to 500 events per request with path, userAgent and optional ip, referrer, hostname and timestamp per event. See the Stats API reference for the full schema.

Verified vs claimed

Anyone can send a fake GPTBot user-agent, so the report distinguishes:

A spoofed user-agent is harmless: it is stored as claimed, never verified, and stays out of your human metrics either way.

Which crawlers the snippets match

The snippets above forward: GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, anthropic-ai (Anthropic); PerplexityBot; Google-Extended; CCBot (Common Crawl); Bytespider (ByteDance); Amazonbot; and Meta-ExternalAgent. Numative classifies whatever reaches it, so you can widen the regex if you want search engine bots from your origin too; the log-drain path already sees everything.