Bot Detection

JA4 + JA3 fingerprints, header anomaly checks, browser-API probes. Action: log, challenge, or block.

What it catches

Three categories of automated traffic, each with different signatures:

  • Naive bots — curl, wget, requests/python-requests, no headers, default UA. Trivial to detect.
  • Scripted bots — headless Chrome, Puppeteer, Playwright. Browser-real UA but missing the runtime quirks of a real browser.
  • Fingerprint-tuned bots — bots that have specifically randomized JA4, rotated UAs, and mimic mouse movement. These are the expensive ones to detect; bot detection gets them <90% of the time and the rest fall to Anomaly Scoring.

Detection signals

JA4 TLS fingerprint
JA3 fallback
HTTP/2 SETTINGS order
Header order + casing
UA string parsed vs known bot DB (Mozilla DB + Sectora-curated)
Missing Accept-Language
Cookie behavior (set + return)
JS runtime check (when feasible)
WebDriver/automation API detection

Actions

  • Log — Record but don't block. Recommended for the first 24h.
  • Challenge — Issue a Turnstile or JS challenge. Humans pass transparently; bots fail.
  • Block — Return 403.

Configurable per bot type. Common pattern: Block naive bots, Challenge scripted bots, Logfingerprint-tuned bots until you've tuned thresholds.

Allow-listing good bots

Some bots you want — Googlebot, Bingbot, your own monitoring, payment-provider webhooks. Bot Detection ships with a curated allowlist of verified-good bots (uses reverse DNS validation, not just UA — UA can be spoofed).

  • Googlebot, Bingbot, DuckDuckBot, Baidubot, YandexBot
  • Stripe / Twilio / Slack / PagerDuty webhooks (verified by signing header)
  • UptimeRobot / Pingdom / StatusCake / etc.
  • Your own monitoring — add via Settings → Shield → Allow patterns.

Tuning

Bot detection is the most aggressive Shield layer. False positives = real users locked out. Tune cautiously:

  • Start in Log mode for at least a week. Review what would have been blocked.
  • Add allow patterns for any partner/monitor false positives.
  • Move to Challenge first. If the challenge solve rate is > 90% for would-be blocks, your tuning is right; if < 50%, you're catching real bots.
  • Move to Block only for high-confidence signatures.

What's next