Skip to main content
OpenAIAnthropicGooglePerplexityMetaMicrosoftAmazonCohere
8 AI crawlers logged in real time
ai bot tracking

Every AI crawler hit, in one live dashboard.

GPTBot. ClaudeBot. PerplexityBot. Google-Extended. Bytespider. Every hit shows up with timestamp, URL, status and user-agent — the moment it lands.

Start free trialSee pricing

free trial · live logs from minute one

liveYovale AI Bot Tracking
yovale.com/en/features/ai-bot-tracking
live crawler feed

Watch GPTBot read your pricing page.

The crawler feed updates as hits arrive. Filter by bot, URL or status code. Export to CSV when a stakeholder wants proof.

GPTBot / 24h
1,284+12%
ClaudeBot / 24h
618+18%
PerplexityBot / 24h
402+7%
Google-Extended / 24h
289+4%
recent crawls
live
  • 12:04:22OpenAIGPTBot/pricing200
  • 12:04:18AnthropicClaudeBot/wordpress-hosting200
  • 12:03:55PerplexityPerplexityBot/docs/backups200
  • 12:03:42GoogleGoogle-Extended/200
  • 12:03:11ByteDanceBytespider/features/woocommerce200
8
AI bots tracked individually
<1s
log lag from hit to dashboard
90 days
retention, per site
the gap

Most hosts hide AI traffic in raw nginx logs.

×GA4freedoesn't log bot traffic
×Plausible$9/mointentionally excludes bots
×nginx access.logssh requiredgrep, sort, uniq, repeat

AI crawlers don't show up in Google Analytics. They don't show up in Plausible. They don't show up in Jetpack. They show up in /var/log/nginx/access.log, mixed in with every favicon request and CSS asset.

By the time you ssh in and grep for ClaudeBot, the conversation is over. The crawler has moved on. You have no idea which pages it read or whether it got 200s or 500s.

Yovale streams crawler hits to the dashboard as they happen. Per-bot panels, per-URL breakdowns, status-code filters — the kind of view a hosting product should have shipped years ago.

Start free trial
$149
/year
capabilities

Crawler observability, built in.

Four primitives that make AI traffic legible. All four ship with every Yovale plan.

  • 01
    01
    real time

    Live hit stream

    Server-sent events push every crawler hit to the dashboard within a second. Page stays open, hits scroll in.

    <1s log lag
  • 02
    02
    segmentation

    Per-bot panels

    Each of the 8 known AI bots gets its own panel — hit count, top URLs, status code distribution, user-agent string.

    8 dedicated panels
  • 03
    03
    drill down

    Per-URL bot view

    Open any URL and see which AI bots have read it, when, and what they got back. Useful for finding 404s only crawlers hit.

    URL-level breakdown
  • 04
    04
    ops

    CSV export and webhooks

    Pipe hits to a webhook for downstream dashboards, or export 90 days of raw events to CSV when an SEO team asks.

    webhook + CSV
side by side

What real AI bot visibility looks like.

Analytics / SaaS tools

Bot traffic captured
excluded by default
Lag from hit to view
5-10 min batched
Per-URL breakdown
aggregate only
Status code visibility
not surfaced
Retention
varies by plan
Cost
$9-$200/mo extra
Result — blind to AI

Yovale built in

Bot traffic captured
8 bots tracked individually
Lag from hit to view
<1 second SSE stream
Per-URL breakdown
URL-level, per-bot
Status code visibility
200, 301, 404, 5xx filtered
Retention
90 days at every tier
Cost
included with hosting
Result — live and labelled

bot tracking stack · how it works

From nginx log line to dashboard panel.

Crawler hits are parsed at the edge, classified by user-agent, deduplicated, and pushed to your dashboard over a server-sent events stream. Nothing to install, nothing to maintain.

Capture
Classify
Stream
1
log stream
Capture
nginx + log parser

Every request hits the nginx access log. A sidecar parser reads the tail, classifies bot user-agents and tags AI crawlers with a canonical name.

8
bots known
Classify
user-agent registry

8 known AI bots are matched by exact UA string. Unknown bots are kept as 'other AI' until we ship support. False positives are rare.

<1s
log lag
Stream
SSE → dashboard

Classified hits flow to the dashboard via server-sent events. The page stays open, hits scroll in, charts re-render. No polling, no refresh.

live in dashboard
pricing

Bundled with hosting, not sold separately.

We don't gate bot tracking behind a higher tier. Starter, Growth, Business — all three see every AI crawler hit, all the time.

$149
/year

Starter, 2 sites, full crawler dashboard on both. Annual only.

8 AI bots tracked<1s log lag90 day retentionPer-URL breakdownCSV exportWebhook delivery
Start free trial
internal links

What pairs with bot tracking.

AI visibility
Make sure bots actually find content worth tracking.
AI referral tracking
From bot hits to humans who came from AI chat.
AI analytics
Charts and trends on top of the raw event stream.
Web intelligence
Schema, freshness, crawl health alongside bot hits.
Performance
Fast TTFB means more pages crawled per visit.
related features

Goes well with these.

WordPress hosting
Docker-isolated PHP-FPM serves clean logs to the parser.
Resource monitoring
Same dashboard, same SSE stream, different signal.
Security
Tell bots apart from attackers without blocking the good ones.
Pricing
Every plan includes the full crawler dashboard.
questions

Things buyers ask before trusting the bot dashboard.

Which AI bots are tracked?+

GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Meta-ExternalAgent, Bytespider, Amazonbot and Applebot-Extended. Unknown bots are kept under 'other AI' until we add explicit support.

Will tracking slow down my site?+

No. We parse the nginx access log tail in a sidecar process. The web request path is untouched — same nginx, same PHP-FPM, same response times.

Does this work behind Cloudflare?+

Yes. Cloudflare passes the real user-agent through. We classify on the origin so even bots that get cached at the edge still show up when they revalidate.

Can I see hits for a specific URL?+

Yes. Click any URL in the dashboard and you get a chronological list of bot hits, with bot name, status code and response time.

Is there an API?+

Webhook out for new hits, REST endpoint for historical queries. Both authenticated with a per-site token. CSV export covers the no-code case.

How long is data retained?+

90 days, every tier. CSV export available at any point in that window.

closing

Stop guessing whether AI reads your site.

Move a site over in seven minutes. By the time you've made coffee, the crawler dashboard will have its first real data points.

Start free trial
free trial · live logs from minute one