AI crawler documentation shifted toward IP verification, bot-specific UA disclosure, and infrastructure monetization

Over the past year, AI crawler documentation has undergone four major structural shifts. First, leading vendors—Anthropic, Amazon, Google, and OpenAI—have published or clarified canonical IP-range lists, enabling site operators to verify crawler authenticity via network-layer detection rather than User-Agent matching alone. Second, crawler UA strings have proliferated and become more granular: OpenAI disclosed OAI-AdsBot for ad-submission traffic, Amazon split Amazonbot into three distinct agents with independent AI-training semantics, and CCBot's canonical string was formalized with a trailing URL. Third, page-level opt-out mechanisms (Amazon's `noarchive` meta tag, Cloudflare's canonical redirects and pay-per-crawl) have emerged alongside robots.txt, widening the toolkit available to publishers. Fourth, Cloudflare's AI Crawl Control and Radar suite have matured into operational platforms—adding content-format diagnostics, WAF rule layering, and public visibility metrics for adoption of llms.txt and AI-specific robots.txt directives—suggesting crawler policy enforcement is becoming a managed service rather than a DIY configuration task.

• IP verification expansion
• UA string proliferation
• Multi-layer opt-out mechanisms
• Managed crawler infrastructure
• Publisher monetization models

Synthesized by Claude Haiku 4.5 from the last 365 days of detected events in this pillar. Regenerates each daily run. Methodology.