Cloudflare AI Search Adds Custom Domains, MCP Access Controls, and Link-Discovery Crawling

Summary

Cloudflare has updated its AI Search service to support serving search endpoints from custom domains with Cloudflare Access authentication, enabling controlled access to agent endpoints like /mcp. The release also introduces namespace-level multi-instance fanout and a new discover parse type that crawls websites by following links rather than relying solely on sitemaps.

Insight

By bundling link-depth web crawling with gated Model Context Protocol (/mcp) endpoints behind identity tokens, infrastructure providers are standardizing the full pipeline from raw site indexing to secure agent retrieval.

Implication

Website owners and enterprise developers can restrict automated AI agent retrieval to authorized consumers while deploying custom crawlers that index sites lacking structured sitemaps.

Why it matters

As websites transition from static search indexes to agent-facing endpoints, native crawler discovery combined with zero-trust access management ensures organizations can govern how their content is surfaced and queried by automated agents.

Evidence

• Primary source
• Evidence ID: cf-changelog-ai-search--d0c79db373ba0a55