Cloudflare AI Crawl Control adds Content Format insights and renames Robots.txt tab to "Directives"
Source: https://developers.cloudflare.com/changelog/post/2026-04-17-tools-for-agentic-internet/
What changed
Cloudflare’s AI Crawl Control changelog documents two new additions: (1) a Content Format chart in the Metrics tab showing what content types AI systems request vs. what the origin serves; (2) the Robots.txt tab has been renamed to Directives and now includes a link to the third-party Agent Readiness score checker. Both changes are framed around readiness for an “agentic Internet” where AI agents are treated as first-class web citizens.
Implication
Site operators using Cloudflare’s AI Crawl Control now have a new signal (Content Format chart) to diagnose mismatches between what AI crawlers request and what their origin delivers, and a renamed UI surface (“Directives”) that broadens scope beyond robots.txt alone — suggesting Cloudflare intends to expand agent-specific crawl controls further. The tie-in to isitagentready.com and the accompanying blog post indicates Cloudflare is actively positioning itself as an infrastructure layer for the AI agent ecosystem.