Crawler type: html_page developers.google.com↗

Google

stable for 1 days · 2 material events tracked · 2 snapshots in history

Documented user-agents (11)

Each distinct UA this vendor publishes on its docs page, extracted by Haiku from the latest snapshot. New UAs appearing or scope changes here are the high-signal events to watch.

User-agent Purpose Scope / when it fires Opt-out
Googlebot Building Google Search index, Discover, Google Images, Video, and News Crawls general web content for search indexing across multiple products User-agent: Googlebot / Disallow: / in robots.txt
Googlebot-Image Crawling images for Google Images, Discover, Video, and search features Crawls image URLs specifically for image-related search products User-agent: Googlebot-Image / Disallow: / in robots.txt
Googlebot-Video Crawling videos for video-related Google Search features Crawls video content for video search and related products User-agent: Googlebot-Video / Disallow: / in robots.txt
Googlebot-News Crawling content for Google News product and news.google.com Crawls news content using various Googlebot user agent strings User-agent: Googlebot-News / Disallow: / in robots.txt
Storebot-Google Crawling for Google Shopping and shopping-related search features Crawls product information for Google Shopping surfaces User-agent: Storebot-Google / Disallow: / in robots.txt
Google-InspectionTool Testing for Rich Result Test and URL inspection in Google Search Console Crawls for search testing and validation tools only User-agent: Google-InspectionTool / Disallow: / in robots.txt
GoogleOther Generic crawling for various Google product teams and internal R&D Generic crawling for research, development, and product-specific needs User-agent: GoogleOther / Disallow: / in robots.txt
GoogleOther-Image Fetching publicly accessible image URLs for various Google product teams Image-specific version of GoogleOther for product team use User-agent: GoogleOther-Image / Disallow: / in robots.txt
GoogleOther-Video Fetching publicly accessible video URLs for various Google product teams Video-specific version of GoogleOther for product team use User-agent: GoogleOther-Video / Disallow: / in robots.txt
Google-CloudVertexBot Crawling for building Vertex AI Agents as requested by site owners Crawls content for Vertex AI Agent construction by site owner request User-agent: Google-CloudVertexBot / Disallow: / in robots.txt
Google-Extended Training Gemini models and providing grounding for Gemini Apps and Vertex AI Controls whether crawled content may be used for Gemini model training and grounding User-agent: Google-Extended / Disallow: / in robots.txt
Change timeline — diffs over time with insights

Each block is a detected change: the new-vs-prior snapshot diff and the LLM-written insight. Newest first.

2025-10-28 → 2026-04-19 173 days apart
+2 −2
IP range JSON source renamed from googlebot.json to common-crawlers.json
material importance 0.70
View diff 
Index: google-extended
===================================================================
--- google-extended	2025-10-28
+++ google-extended	2026-04-19
@@ -9,7 +9,7 @@
 technical properties
 of Google's crawlers also apply to the common crawlers.
 The common crawlers generally crawl from the IP ranges published in the
-googlebot.json
+common-crawlers.json
 object, and the reverse DNS mask
 of their hostname matches
 crawl-***-***-***-***.googlebot.com
@@ -296,4 +296,4 @@
 . For details, see the
 Google Developers Site Policies
 . Java is a registered trademark of Oracle and/or its affiliates.
-Last updated 2025-04-25 UTC.
\ No newline at end of file
+Last updated 2026-02-11 UTC.
\ No newline at end of file
Events