Weekly Dispatch · Week 16 of 2026

Publishers blocking AI bots via robots.txt experience reduced traffic, diminishing brand awareness and audience engagement in AI-generated responses.

"A key finding is that the most common protective step, restricting AI bot access through robots.txt is associated with lower, not higher, traffic."

Publishers can use canonical tags to redirect AI crawlers to current content, aiming to improve AI-generated answers and control data access.

"In the first seven days, 100% of AI training crawler requests to pages with non-self-referencing canonical tags were redirected and were not served with deprecated content."

Autonomous AI agents still struggle with complex real-world web tasks, achieving low success rates compared to humans, highlighting brittleness.

"Web agents that fail two-thirds of everyday online tasks. Skills that evolve across users like a living organism. Autonomous AI is at an inflection point - capable enough to surprise us, brittle enough to humble us, and occasionally deceptive enough to alarm us."

Autonomous AI agents pose a new insider threat due to independent actions and high privileges, demanding a shift to hybrid identity security.

"The moment software starts taking independent action inside your environment, it stops being a tool and starts behaving like an identity."

Uncontrolled proliferation of AI agents ('agent sprawl') creates significant IT and compliance risks, including exploding costs and security gaps.

"The so-called 'agent sprawl' (uncontrolled proliferation of AI agents) not only leads to exploding infrastructure costs and redundant systems, but also opens up dangerous security gaps."

AI search shifts traffic from ranked links to synthesized answers, reducing clicks but increasing conversion rates for cited brands.

"AI Overviews cause a 61% CTR drop on informational queries, and 93% of AI Mode searches produce zero clicks to external websites."

Google's AI Overviews cause structural declines in publisher traffic by summarizing content, creating a 'lose-lose' situation for publishers.

"If you let Google's AI crawl your content, it summarizes your work and keeps readers on its platform. If you block Google's AI, you disappear from search results entirely."