AI crawler documentation converged on IP verification, opt-out granularity, and infrastructure-level routing controls

Over the past year, AI crawler documentation has shifted decisively toward operational maturity: vendors have published IP-range lists (Anthropic, Amazon, OpenAI, Google), clarified and expanded user-agent strings to disambiguate search vs. training vs. specialized roles (Amazon added three distinct bots; OpenAI disclosed OAI-AdsBot for the first time), and reframed opt-out mechanisms from binary robots.txt blocks to granular page-level tags and infrastructure redirects. Simultaneously, platform providers—most prominently Cloudflare—have layered canonical-redirect and content-format verification features that allow publishers to steer AI training crawlers without affecting search traffic, signaling a shift from blocking to routing. The adoption of emerging standards like llms.txt by Perplexity and Cloudflare's visibility tooling suggest the ecosystem is moving toward declarative, machine-readable site postures. These changes address three converging pressures: firewall operators demand IP ranges for accurate verification; publishers demand fine-grained controls to monetize or selectively block; and crawler vendors demand clearer signals (llms.txt, directives tabs) to avoid over-crawling or missing content.

• IP range publication expansion
• Granular opt-out mechanisms
• UA string disambiguation
• Infrastructure-level routing controls
• Machine-readable site standards

Synthesized by Claude Haiku 4.5 from the last 365 days of detected events in this pillar. Regenerates each daily run. Methodology.