stable for 49 days · 2 material events tracked · 1 snapshot in history
Each distinct UA this vendor publishes on its docs page, extracted by Haiku from the latest snapshot. New UAs appearing or scope changes here are the high-signal events to watch.
| User-agent | Purpose | Scope / when it fires | Opt-out |
|---|---|---|---|
AI Training Crawlers (Generic) | Training data collection for AI models | General web crawl for machine learning dataset generation | Redirects for AI Training feature and robots.txt disallow rules |
Verified AI Crawlers | Authenticated content access with potential monetization | Crawl-controlled content with payment via HTTP 402 responses | HTTP 402 Payment Required responses; robots.txt compliance honored |
Agentic Crawlers/Browsers | Autonomous agent automation and interaction | General web interaction for multi-agent orchestration systems | Agent Readiness Score controls and robots.txt disallow |
The crawler insights digest was substantially refreshed. Eight previous items were removed entirely: dedicated AI training crawlers approaching 50% of bot traffic; publisher struggles with the third-party scraper economy
This source went from empty to a 14-item digest covering the AI crawler and bot-policy landscape through mid-April 2026. Key hard facts include: (1) [Cloudflare's "Redirects for AI Training"](https://blog.cloudflare.com/