Cloudflare AI Search Adds Custom Domains, MCP Access Controls, and Link-Discovery Crawling

Development

Cloudflare has updated its AI Search service to support serving search endpoints from custom domains with Cloudflare Access authentication, enabling controlled access to agent endpoints like /mcp. The release also introduces namespace-level multi-instance fanout and a new discover parse type that crawls websites by following links rather than relying solely on sitemaps.

Why it matters

As websites transition from static search indexes to agent-facing endpoints, native crawler discovery combined with zero-trust access management ensures organizations can govern how their content is surfaced and queried by automated agents.

Trend impact

• agent-retrieval-infrastructure
• mcp-access-control
• managed-web-crawling

Evidence

• Primary source
• Evidence ID: cf-changelog-ai-search--d0c79db373ba0a55