https://crawlerpolicy.com Daily verified developments, reported intelligence, and early signals across the machine-readable web. en-US Tue, 08 Sep 2026 12:40:20 GMT https://crawlerpolicy.com/events/applebot-extended-apple-clarifies-applebot-extended-opt-outs-do-not-affect-search-rankings https://crawlerpolicy.com/events/applebot-extended-apple-clarifies-applebot-extended-opt-outs-do-not-affect-search-rankings Sat, 05 Sep 2026 11:40:15 GMT crawler-controls crawler-controls search-discovery Apple Apple updated its Applebot support documentation to confirm that website rules applied to Applebot-Extended are not factored into Apple Search ranking. The update provides explicit assurance that disallowing the AI training crawler in robots.txt will not impact organic visibility across Spotlight, Siri, and Safari. Why it matters: As platforms expand generative AI features, web publishers frequently worry that opting out of model training may invisibly penalize their search distribution. Apple's documented commitment ensures crawler governance decisions remain strictly partitioned from search ranking. https://crawlerpolicy.com/events/ietf-webbotauth-wg-ietf-web-bot-auth-wg-officially-adopts-http-message-signatures-protocol-draft https://crawlerpolicy.com/events/ietf-webbotauth-wg-ietf-web-bot-auth-wg-officially-adopts-http-message-signatures-protocol-draft Wed, 02 Sep 2026 12:35:00 GMT standards-protocols standards-protocols crawler-controls agentic-web IETF Web Bot Auth Working Group The IETF Web Bot Auth Working Group has formally adopted the core specification for cryptographically signing automated web requests. The individual submission was re-indexed as the working group document draft-ietf-webbotauth-httpsig-protocol-00 ('HTTP Message Signatures for automated traffic') on September 1, 2026. This marks the transition of the protocol from an individual draft into the working group's official standards track. Why it matters: As web publishers struggle with scraping volume and bot spoofing, a standardized cryptographic authentication protocol under active IETF development provides a path toward verifiable, accountable crawler identification at scale. https://crawlerpolicy.com/events/web-bot-auth-repo-cloudflare-optimizes-ed25519-signature-verification-for-web-bot-auth-edge-worklo https://crawlerpolicy.com/events/web-bot-auth-repo-cloudflare-optimizes-ed25519-signature-verification-for-web-bot-auth-edge-worklo Mon, 31 Aug 2026 16:22:52 GMT standards-protocols standards-protocols agentic-web crawler-controls Web Bot Auth (spec repo) A pull request to Cloudflare's Web Bot Auth reference implementation optimizes cryptographic signature checking by precomputing and caching prepared Ed25519 verifying keys when keyrings are populated. The change eliminates redundant point decompression on every verification request while keeping the public API and error handling intact. Local benchmarks indicate an approximate 10% reduction in verification overhead, saving roughly 2.1 microseconds per request. Why it matters: As cryptographic bot verification moves into high-volume edge infrastructure, micro-optimizations in key parsing and signature evaluation are necessary to make cryptographic crawler verification practical at production scale. https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-adaptive-intelligence-bot-detection-engine https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-adaptive-intelligence-bot-detection-engine Mon, 31 Aug 2026 12:59:00 GMT crawler-controls crawler-controls measurement-economics Cloudflare Blog Cloudflare has introduced Adaptive Intelligence, a continuous-learning bot detection engine integrated into its Bot Management platform. The initial rollout replaces fixed, scheduled machine learning releases with live-traffic model retraining that measures automated abuse probabilities per request. Future components will generate ephemeral, disposable detection rules and incorporate cross-network customer feedback signals. Why it matters: As automated scraping tools become more agile, CDN-level bot management is shifting from static perimeter rules to continuous adversarial learning, fundamentally changing the economic viability of unauthorized automated web extraction. https://crawlerpolicy.com/events/eu-commission-digital-european-commission-designates-chatgpt-as-vlose-reddit-and-roblox-as-vlops-under https://crawlerpolicy.com/events/eu-commission-digital-european-commission-designates-chatgpt-as-vlose-reddit-and-roblox-as-vlops-under Mon, 31 Aug 2026 07:28:03 GMT policy-regulation policy-regulation search-discovery EU Commission Press The European Commission has officially designated ChatGPT as a Very Large Online Search Engine (VLOSE) under the Digital Services Act (DSA). In the same regulatory decision, the Commission also designated Reddit and Roblox as Very Large Online Platforms (VLOPs). Why it matters: Classifying generative AI chat tools as search engines establishes a major regulatory precedent that extends European search governance directly into the operations of AI query retrieval and indexing systems. https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-botbase-for-operators-with-automated-verification-and-conten https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-botbase-for-operators-with-automated-verification-and-conten Fri, 28 Aug 2026 12:59:44 GMT crawler-controls crawler-controls standards-protocols agentic-web Cloudflare Blog Cloudflare has introduced BotBase for Operators, a dedicated dashboard interface enabling automated crawler and agent operators to register bots, track submission statuses, and edit technical configurations. The intake process adopts an updated taxonomy requiring operators to classify their primary behaviors, operational model (direct operator versus intermediary infrastructure), and intended content usage matching the Content Signals framework. To handle submission volumes that increased sevenfold since 2023, Cloudflare also automated technical validation of user-agent uniqueness, IP lists, reverse DNS, and Web Bot Auth cryptographic signatures. Why it matters: As automated web traffic grows, manual allowlisting and simple user-agent verification are no longer sustainable. Centralizing verification, taxonomy, and content-use declarations inside the CDN layer establishes an enforceable baseline for how automated agents interact with publishers. https://crawlerpolicy.com/events/datadome-blog-datadome-launches-hosted-model-context-protocol-server-for-bot-telemetry https://crawlerpolicy.com/events/datadome-blog-datadome-launches-hosted-model-context-protocol-server-for-bot-telemetry Thu, 27 Aug 2026 13:10:37 GMT agentic-web agentic-web standards-protocols crawler-controls DataDome Blog DataDome introduced a hosted MCP server that connects its Trend Reports telemetry directly to AI assistants and autonomous workflows. The integration allows teams to authenticate with existing DataDome credentials and query scraping activity, credential stuffing metrics, and bot traffic spikes using natural language via the Model Context Protocol (MCP). The service is hosted by DataDome as part of its Agent Trust framework. Why it matters: As bot management intersects with agentic web access, standard protocols like MCP allow organizations to automate how they monitor, analyze, and react to automated scraping and malicious bot traffic in real time. https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-bot-preference-sync-to-automatically-align-robots-txt-with-e https://crawlerpolicy.com/events/cloudflare-blog-cloudflare-launches-bot-preference-sync-to-automatically-align-robots-txt-with-e Fri, 21 Aug 2026 23:19:57 GMT crawler-controls crawler-controls search-discovery agentic-web standards-protocols Cloudflare Blog Cloudflare has introduced Bot Preference Sync, a feature across all plan tiers that automatically prepends and maintains robots.txt directives corresponding to a site owner's dashboard configurations for AI Search, Agent, and Training traffic. The system pairs edge-enforced blocks with matching robots.txt rules drawn from its tracked bot directory, and introduces an onboarding default for ad-supported publishers that disallows training while permitting search indexing. Additionally, Cloudflare established stricter verification criteria for mixed-use crawlers, requiring them to respect no-training signals, allow opt-outs from AI summaries, provide page-level training visibility, and prove search indexing parity. Why it matters: Discrepancies between robots.txt files and edge firewall policies have historically provided AI crawlers technical or legal ambiguity to ignore site preferences; synchronizing them at the CDN layer standardizes machine-readable compliance enforcement at web scale. https://crawlerpolicy.com/events/web-bot-auth-repo-web-bot-auth-rust-verifier-hardened-against-future-dated-signatures https://crawlerpolicy.com/events/web-bot-auth-repo-web-bot-auth-rust-verifier-hardened-against-future-dated-signatures Tue, 18 Aug 2026 02:19:46 GMT standards-protocols standards-protocols crawler-controls agentic-web Web Bot Auth (spec repo) A pull request to Cloudflare's Web Bot Auth specification repository updates the Rust verifier implementation to fail closed when a signature's created timestamp is in the future. Previously, the Rust library verified the cryptographic signature before checking created timestamps, diverging from the TypeScript reference implementation which already rejected them upfront. The change aligns both implementations by validating timestamp windows prior to running cryptographic checks. Why it matters: As cryptographic bot verification frameworks mature into industry standards, consistent fail-closed behavior across reference libraries prevents implementation-specific security bypasses and replay vulnerabilities. https://crawlerpolicy.com/events/web-bot-auth-repo-cloudflare-patches-web-bot-auth-rust-verifier-to-enforce-expiration-checks https://crawlerpolicy.com/events/web-bot-auth-repo-cloudflare-patches-web-bot-auth-rust-verifier-to-enforce-expiration-checks Thu, 13 Aug 2026 03:22:43 GMT standards-protocols standards-protocols crawler-controls agentic-web Web Bot Auth (spec repo) Cloudflare updated its Web Bot Auth Rust reference implementation to reject expired signatures by default during verification. Previously, `WebBotAuthVerifier::verify` permitted signatures with elapsed expiration timestamps if the cryptographic payload was otherwise valid. The change aligns the Rust library's fail-closed behavior with the TypeScript implementation. Why it matters: As cryptographic signatures become the cornerstone for verifying automated agents, inconsistent expiration validation across SDKs could allow malicious actors to exploit stale credentials. https://crawlerpolicy.com/events/datadome-blog-forrester-expands-bot-management-category-to-bot-and-agent-trust-management https://crawlerpolicy.com/events/datadome-blog-forrester-expands-bot-management-category-to-bot-and-agent-trust-management Mon, 10 Aug 2026 22:00:03 GMT agentic-web agentic-web crawler-controls measurement-economics DataDome Blog Analyst firm Forrester updated its market category to Bot and Agent Trust Management Software for Q2 2026, shifting focus from binary bot blocking to managing human-initiated AI agents and traffic intent. In an interview with DataDome, report author Sandy Carielli explained that generalist CDN and WAF vendors were excluded from the evaluation due to lack of deep agentic signal depth. Over half of surveyed enterprise buyers reported actively investigating AI agent trust use cases, driven by top-line business impacts such as LLM search indexing and automated B2B procurement. Why it matters: As user agents increasingly act on behalf of paying consumers and business buyers, blunt crawler blocking risks choking discovery and transaction revenue rather than merely mitigating operational cost. https://crawlerpolicy.com/events/datadome-blog-datadome-publishes-ai-crawler-taxonomy-and-verification-guidance-across-2026-bot https://crawlerpolicy.com/events/datadome-blog-datadome-publishes-ai-crawler-taxonomy-and-verification-guidance-across-2026-bot Thu, 06 Aug 2026 12:09:01 GMT crawler-controls crawler-controls agentic-web measurement-economics DataDome Blog Bot management firm DataDome published an operational guide and measurement report analyzing AI bots across three primary functions: training foundation models, indexing for search/grounding, and live-fetching content at user request. The report highlights empirical tests showing that nearly 80% of 698,214 analyzed websites allowed spoofed ChatGPT-User requests through due to lack of IP verification. It details operator policies where live fetchers from OpenAI, Meta, and Perplexity may bypass or ignore robots.txt directives, urging publishers to adopt multi-tiered technical controls. Why it matters: As AI search and training split into separate technical pipelines, publishers face asymmetric economic costs—serving billions of unmonetized compute-heavy requests from companies like Meta with negligible referral value—making active request verification essential for traffic control. https://crawlerpolicy.com/events/cf-changelog-ai-search-cloudflare-ai-search-adds-custom-domains-mcp-access-controls-and-link-discovery- https://crawlerpolicy.com/events/cf-changelog-ai-search-cloudflare-ai-search-adds-custom-domains-mcp-access-controls-and-link-discovery- Thu, 06 Aug 2026 00:00:00 GMT agentic-web agentic-web search-discovery crawler-controls Cloudflare AI Search changelog Cloudflare has updated its AI Search service to support serving search endpoints from custom domains with Cloudflare Access authentication, enabling controlled access to agent endpoints like `/mcp`. The release also introduces namespace-level multi-instance fanout and a new `discover` parse type that crawls websites by following links rather than relying solely on sitemaps. Why it matters: As websites transition from static search indexes to agent-facing endpoints, native crawler discovery combined with zero-trust access management ensures organizations can govern how their content is surfaced and queried by automated agents. https://crawlerpolicy.com/events/datadome-blog-datadome-outlines-framework-for-non-human-identity-and-ai-agent-access-control https://crawlerpolicy.com/events/datadome-blog-datadome-outlines-framework-for-non-human-identity-and-ai-agent-access-control Sat, 01 Aug 2026 12:39:34 GMT agentic-web agentic-web crawler-controls standards-protocols DataDome Blog DataDome published an architectural overview analyzing why traditional Identity and Access Management (IAM) fails for autonomous AI agents and Model Context Protocol (MCP) endpoints. The security vendor highlights that static access reviews and role-based permissions cannot govern non-human identities that extend permissions mid-task or chain external tools. To address agent sprawl, the post advocates layering dynamic authorization models with real-time behavioral monitoring and identity verification. Why it matters: As enterprises scale agentic deployments, relying on self-declared user-agent strings or static OAuth tokens exposes internal databases and APIs to severe supply-chain compromises. Effective bot and crawler controls must evolve beyond simple perimeter blocklists to fine-grained, intent-aware authorization. https://crawlerpolicy.com/events/cf-changelog-ai-search-cloudflare-adds-ai-search-integrations-for-vercel-ai-sdk-langchain-and-agents-sd https://crawlerpolicy.com/events/cf-changelog-ai-search-cloudflare-adds-ai-search-integrations-for-vercel-ai-sdk-langchain-and-agents-sd Thu, 30 Jul 2026 00:00:00 GMT agentic-web agentic-web search-discovery Cloudflare AI Search changelog Cloudflare has introduced direct integrations for its AI Search product across major agent development frameworks. The release includes a new ai-search-provider package for the Vercel AI SDK, a dedicated retriever in the langchain-cloudflare library, and integration guides for the Cloudflare Agents SDK. Why it matters: As web discovery shifts from manual browsing to agent-driven ingestion, standardized SDK interfaces determine how easily synthetic agents can access, cite, and retrieve indexed content. https://crawlerpolicy.com/events/mcp-spec-model-context-protocol-issues-stable-2026-07-28-specification-release https://crawlerpolicy.com/events/mcp-spec-model-context-protocol-issues-stable-2026-07-28-specification-release Tue, 28 Jul 2026 16:47:49 GMT standards-protocols standards-protocols agentic-web Model Context Protocol (spec) The Model Context Protocol project has published the stable 2026-07-28 revision of its specification. The update and detailed documentation are hosted on the official Model Context Protocol website. Implementers can review specific updates in the 2026-07-28 changelog. Why it matters: Standardized protocols are essential for ensuring interoperable data access and execution boundaries across emerging AI agent ecosystems. https://crawlerpolicy.com/events/news-media-alliance-bipartisan-u-s-house-bill-targets-undisclosed-ai-stealth-crawlers https://crawlerpolicy.com/events/news-media-alliance-bipartisan-u-s-house-bill-targets-undisclosed-ai-stealth-crawlers Thu, 23 Jul 2026 18:59:23 GMT policy-regulation policy-regulation crawler-controls News/Media Alliance U.S. Representatives Laurel Lee and Valerie Foushee introduced the Stealth Bot Prohibition Act to enforce crawler transparency. The bipartisan legislation aims to require AI crawlers that obscure their origin to disclose their identity and purpose to web operators. Why it matters: If passed, statutory identification mandates would make crawler impersonation and stealth data harvesting legally actionable rather than just a violation of site terms or robots.txt conventions. https://crawlerpolicy.com/events/web-bot-auth-repo-web-bot-auth-fixes-signature-verification-bypass-allowing-request-replay https://crawlerpolicy.com/events/web-bot-auth-repo-web-bot-auth-fixes-signature-verification-bypass-allowing-request-replay Tue, 21 Jul 2026 09:35:47 GMT standards-protocols standards-protocols crawler-controls agentic-web Web Bot Auth (spec repo) A vulnerability in the Web Bot Auth implementation allowed signatures with empty component lists to pass verification without binding to a request target. Pull request #114 resolves security advisory GHSA-x9cc-346q-g27m by requiring signature coverage of @authority or @target-uri and signature-agent headers. The fix ensures verification binds signatures to specific HTTP request targets, matching the behavior of the Rust reference implementation. Why it matters: Emerging bot authentication standards depend entirely on request-bound cryptographic proof, making component verification enforcement essential to preventing unauthorized automated traffic spoofing. https://crawlerpolicy.com/events/cf-changelog-ai-gateway-cloudflare-adds-user-agent-logging-and-filtering-to-ai-gateway https://crawlerpolicy.com/events/cf-changelog-ai-gateway-cloudflare-adds-user-agent-logging-and-filtering-to-ai-gateway Fri, 12 Jun 2026 00:00:00 GMT agentic-web agentic-web crawler-controls standards-protocols Cloudflare AI Gateway changelog Cloudflare has updated AI Gateway to capture the client user agent for every incoming request. Operators can now view the user agent alongside existing request metadata and filter logs in the dashboard by exact match, exclusion, or substring. Why it matters: As automated agents and diverse AI client libraries proliferate, visibility into the exact client software initiating model queries becomes essential for governance, debugging, and traffic attribution. https://crawlerpolicy.com/events/openai-blog-openai-plans-acquisition-of-ona-to-power-persistent-cloud-environments-for-codex https://crawlerpolicy.com/events/openai-blog-openai-plans-acquisition-of-ona-to-power-persistent-cloud-environments-for-codex Thu, 11 Jun 2026 00:00:00 GMT agentic-web agentic-web OpenAI News OpenAI has announced plans to acquire Ona to expand its Codex platform with secure, persistent cloud environments. The infrastructure addition is designed to facilitate long-running AI agents operating across enterprise workflows. Why it matters: Executing complex, multi-step agentic workflows requires dedicated runtime environments rather than isolated completions, making persistent cloud infrastructure a critical layer in enterprise AI agent deployment.