Crawler type: html_page developers.google.com↗

Google

stable for 44 days · 2 material events tracked · 3 snapshots in history

Documented user-agents (11)

Each distinct UA this vendor publishes on its docs page, extracted by Haiku from the latest snapshot. New UAs appearing or scope changes here are the high-signal events to watch.

User-agent Purpose Scope / when it fires Opt-out
Googlebot Build Google Search indexes and power Google Search features General web crawl for Google Search, Discover, Images, Video, News, and related features User-agent: Googlebot / Disallow: / in robots.txt
Googlebot-Image Crawl images for Google Images and related products Image-specific crawling for Google Images, Discover, Video, and Search image features User-agent: Googlebot-Image / Disallow: / in robots.txt
Googlebot-Video Crawl videos for video-related Google Search features Video-specific crawling for video-related Google Search features and dependent products User-agent: Googlebot-Video / Disallow: / in robots.txt
Googlebot-News Crawl content for Google News product News-specific crawling for Google News product, news.google.com, and Google News app User-agent: Googlebot-News / Disallow: / in robots.txt
Storebot-Google Crawl product information for Google Shopping Shopping product crawling for Google Shopping tab and all Google Shopping surfaces User-agent: Storebot-Google / Disallow: / in robots.txt
Google-InspectionTool Power Google Search testing tools and crawl verification Testing-specific crawling for Rich Result Test and URL inspection in Search Console User-agent: Google-InspectionTool / Disallow: / in robots.txt
GoogleOther Generic crawling for various Google product teams and internal R&D General public content fetching for internal research, development, and one-off crawls User-agent: GoogleOther / Disallow: / in robots.txt
GoogleOther-Image Generic image crawling optimized for public image URLs Image-specific generic crawling for various product teams and internal use User-agent: GoogleOther-Image / Disallow: / in robots.txt
GoogleOther-Video Generic video crawling optimized for public video URLs Video-specific generic crawling for various product teams and internal use User-agent: GoogleOther-Video / Disallow: / in robots.txt
Google-CloudVertexBot Crawl content for building Vertex AI Agents requested by site owners Site owner-requested crawling for Vertex AI Agent construction and training User-agent: Google-CloudVertexBot / Disallow: / in robots.txt
Google-Extended Crawling for training Gemini models and grounding in Gemini Apps Content collection for Gemini model training and grounding with Google Search User-agent: Google-Extended / Disallow: / in robots.txt; does not impact Search inclusion or ranking
Change timeline — diffs over time with insights

Each block is a detected change: the new-vs-prior snapshot diff and the LLM-written insight. Newest first.

2026-04-19 → 2026-04-24 5 days apart
+2 −3
View diff 
Index: google-extended
===================================================================
--- google-extended	2026-04-19
+++ google-extended	2026-04-24
@@ -187,8 +187,7 @@
 user agent don't affect
 any specific product. GoogleOther is the generic crawler that may be used by various
 product teams for fetching publicly accessible content from sites. For example, it may
-be used for one-off crawls for internal research and development. It has no effect on
-Google Search or other products.
+be used for one-off crawls for internal research and development.
 GoogleOther-Image
 User-Agent in HTTP requests
 GoogleOther-Image/1.0
@@ -296,4 +295,4 @@
 . For details, see the
 Google Developers Site Policies
 . Java is a registered trademark of Oracle and/or its affiliates.
-Last updated 2026-02-11 UTC.
\ No newline at end of file
+Last updated 2026-04-23 UTC.
\ No newline at end of file
2025-10-28 → 2026-04-19 173 days apart
+2 −2
IP range JSON source renamed from googlebot.json to common-crawlers.json
material importance 0.70
View diff 
Index: google-extended
===================================================================
--- google-extended	2025-10-28
+++ google-extended	2026-04-19
@@ -9,7 +9,7 @@
 technical properties
 of Google's crawlers also apply to the common crawlers.
 The common crawlers generally crawl from the IP ranges published in the
-googlebot.json
+common-crawlers.json
 object, and the reverse DNS mask
 of their hostname matches
 crawl-***-***-***-***.googlebot.com
@@ -296,4 +296,4 @@
 . For details, see the
 Google Developers Site Policies
 . Java is a registered trademark of Oracle and/or its affiliates.
-Last updated 2025-04-25 UTC.
\ No newline at end of file
+Last updated 2026-02-11 UTC.
\ No newline at end of file
Events