Documented user-agents (3)
Each distinct UA this vendor publishes on its docs page, extracted by Haiku from
the latest snapshot. New UAs appearing or scope changes here are the high-signal
events to watch.
| User-agent | Purpose | Scope / when it fires | Opt-out |
ClaudeBot | Collect web content for AI model training and development | General web crawl to gather data for model training datasets | User-agent: ClaudeBot / Disallow: / in robots.txt file; supports Crawl-delay extension |
Claude-User | Support user-initiated web access when Claude AI users ask questions | Retrieves website content in response to user-directed queries to Claude | User-agent: Claude-User / Disallow: / in robots.txt file to prevent retrieval for user requests |
Claude-SearchBot | Improve search result quality and relevance for user search responses | Analyzes online content to enhance relevance and accuracy of search results | User-agent: Claude-SearchBot / Disallow: / in robots.txt file to prevent indexing |
Change timeline — diffs over time with insights
Each block is a detected change: the new-vs-prior snapshot diff and the
LLM-written insight. Newest first.
2026-05-08 → 2026-05-14 6 days apart
+1 −1
View diff
Index: claudebot
===================================================================
--- claudebot 2026-05-08
+++ claudebot 2026-05-14
@@ -48,7 +48,7 @@
Related Articles
Reporting, Blocking, and Removing Content from Claude
How to get support
-Does Anthropic Act as a Data Processor or Controller?
+How can I export my Claude data?
Reporting, Blocking, and Removing Content from Claude
Claude in Chrome Permissions Guide
Did this answer your question?
2026-05-05 → 2026-05-08 3 days apart
+1 −1
View diff
Index: claudebot
===================================================================
--- claudebot 2026-05-05
+++ claudebot 2026-05-08
@@ -1,6 +1,6 @@
Skip to main content
Does Anthropic crawl data from the web, and how can site owners block the crawler?
-Updated over a month ago
+April 7, 2026
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.
Bot
Use
2026-04-28 → 2026-05-05 7 days apart
+1 −1
View diff
Index: claudebot
===================================================================
--- claudebot 2026-04-28
+++ claudebot 2026-05-05
@@ -1,6 +1,6 @@
Skip to main content
Does Anthropic crawl data from the web, and how can site owners block the crawler?
-Updated over 3 weeks ago
+Updated over a month ago
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.
Bot
Use
2026-04-21 → 2026-04-28 7 days apart
+1 −1
View diff
Index: claudebot
===================================================================
--- claudebot 2026-04-21
+++ claudebot 2026-04-28
@@ -1,6 +1,6 @@
Skip to main content
Does Anthropic crawl data from the web, and how can site owners block the crawler?
-Updated over 2 weeks ago
+Updated over 3 weeks ago
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.
Bot
Use
2026-04-19 → 2026-04-21 2 days apart
+1 −1
View diff
Index: claudebot
===================================================================
--- claudebot 2026-04-19
+++ claudebot 2026-04-21
@@ -1,6 +1,6 @@
Skip to main content
Does Anthropic crawl data from the web, and how can site owners block the crawler?
-Updated over a week ago
+Updated over 2 weeks ago
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.
Bot
Use
2025-08-06 → 2026-04-19 256 days apart
+9 −8
View diff
Index: claudebot
===================================================================
--- claudebot 2025-08-06
+++ claudebot 2026-04-19
@@ -1,8 +1,5 @@
Skip to main content
-All Collections
-Privacy & Legal
Does Anthropic crawl data from the web, and how can site owners block the crawler?
-Does Anthropic crawl data from the web, and how can site owners block the crawler?
Updated over a week ago
As per industry standard, Anthropic uses a variety of robots to gather data from the public web for model development, to search the web, and to retrieve web content at users’ direction. Anthropic uses different robots to enable website owner transparency and choice. Below is information on the three robots that Anthropic uses and how to set your site preferences to enable those you want to access your content and limit those you don’t.
Bot
@@ -38,18 +35,22 @@
To block a Bot from your entire website, add this to the robots.txt file in your top-level directory. Please do this for every subdomain that you wish to opt out from. An example of this is:
User-agent: ClaudeBot
Disallow: /
-Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above. Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. Additionally, we do not currently publish IP ranges, as we use service provider public IPs. This may change in the future.
+Opting out of being crawled by Anthropic Bots requires modifying the robots.txt file in the manner above. Alternate methods like blocking IP address(es) from which Anthropic Bots operates may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file. If a crawler has a source IP address on
+this list
+, it indicates that the crawler is coming from Anthropic.
You can learn more about our data handling practices and commitments at our
Help Center
. If you have further questions, or believe that our Bots may be malfunctioning, please reach out to
[email protected]
+[email protected]
. Please reach out from an email that includes the domain you are contacting us about, as it is otherwise difficult to verify reports.
+You can be notified of substantial changes to this article by clicking here and completing the form:
+Subscribe to updates
Related Articles
Reporting, Blocking, and Removing Content from Claude
-How can I access the Anthropic API?
-How to Get Support
-Does Anthropic act as a Data Processor or Controller?
+How to get support
+Does Anthropic Act as a Data Processor or Controller?
Reporting, Blocking, and Removing Content from Claude
+Claude in Chrome Permissions Guide
Did this answer your question?
😞
😐