Preserve what was observed
Every successful fetch can create an immutable evidence record with source identity, timestamps, content hashes, and raw or normalized material. Failed analysis does not discard it.
Method & boundaries
A readable record of the AI × web negotiation.
Crawler Policy turns changes in machine access, content rights, agent identity, governance, and web economics into concise English analysis — while keeping every conclusion tied to replayable evidence.
Editorial flowEvidence → Feed → Development → Weekly
Explain what happened
Gemini produces a structured Summary, Insight, Implication, and Why it matters. Deterministic source rules label the result Verified, Reported, or Signal.
Change status with authority
A durable trend can move only when primary or measurement-tier evidence supports the delta. No-change is a real result.
Synthesize a completed window
The weekly issue compares a full ISO week, shows quiet tracks, and refuses to publish when required health coverage is critical or stale.
Coverage45 configured sources · nine tracks
Stable questions, explicit taxonomy
Sources may cover multiple tracks. Nine internal tracks roll up into five public fronts so the site stays navigable without erasing technical distinctions.
Trust modelWhat the labels mean
Pipeline health is stage-aware
Healthy means all configured sources completed. Degraded means useful output exists but some non-required work failed. Critical means required evidence is incomplete, so weekly conclusions are withheld. Unknown is never promoted to healthy.
Gemini assists; evidence governs
Gemini helps classify relevance, identify material change, and write the four analytical layers. It cannot promote a reported story into verified evidence. Source records, evidence IDs, and raw diffs remain authoritative.
Publication labels have consequences
Verified items use primary or measurement evidence. Reported items come from configured specialist reporting. Signals are early observations. Only verified material items can move trends or weekly conclusions.
Three dates stay separate
event_date is when the development occurred; published_at is when the source published it; detected_at is when this pipeline observed it.
Legacy is not current intelligence
Schema-v1 events and the former Weekly Dispatch are preserved unchanged in Legacy Archive. They remain useful context but never seed current trends or weekly issues.