stable for 1 days · 2 material events tracked · 1 snapshot in history
Each distinct UA this vendor publishes on its docs page, extracted by Haiku from the latest snapshot. New UAs appearing or scope changes here are the high-signal events to watch.
| User-agent | Purpose | Scope / when it fires | Opt-out |
|---|---|---|---|
Agentic crawlers (general) | AI-driven automated tools and agents accessing web content for various purposes | General web crawl across Cloudflare network; now representing ~10% of total requests | Via Cloudflare Agent Readiness controls and redirects for AI training verification; robots.txt supported |
AI training crawlers | Collection of data for AI model training | Content crawling for training datasets; subject to canonical redirects via Redirects for AI Training feature | Publishers can redirect verified crawlers to preferred pages; HTTP 402 Payment Required monetization option available |
The crawler insights digest was substantially refreshed. Eight previous items were removed entirely: dedicated AI training crawlers approaching 50% of bot traffic; publisher struggles with the third-party scraper economy
This source went from empty to a 14-item digest covering the AI crawler and bot-policy landscape through mid-April 2026. Key hard facts include: (1) [Cloudflare's "Redirects for AI Training"](https://blog.cloudflare.com/