Apple

Index: applebot-extended
===================================================================
--- applebot-extended	2026-04-19
+++ applebot-extended	2026-06-09
@@ -1,9 +1,14 @@
 Learn about Applebot, the web crawler for Apple.
-The data crawled by Applebot is used to power various features, such as the search technology that is integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.
+The data crawled by Applebot is used to power various features, such as the search technology integrated into many user experiences in Appleʼs ecosystem including Spotlight, Siri, and Safari. Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.
 The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Web publishers can opt-out from having their content used to train generative foundation models by disallowing
 Applebot-Extended
 in the robots.txt file.
-Even if you disallow Applebot-Extended, your website instructions may still allow Applebot to crawl your webpages. In that case, your content would remain discoverable through Spotlight, Siri, as well as other system-wide features on Apple devices.
+Applebot crawled data may be used to provide additional context and up-to-date content when AI models are used to generate output for display in Apple products and services. For example, answering broad world knowledge questions in Siri and Search that may include links to sources and websites used to help generate the answer. Web publishers can opt out of their content being used in these broad world knowledge answers by applying the
+nosnippet
+meta tag to specific content.
+Even if you disallow Applebot-Extended and tag website content with the
+nosnippet
+meta tag, your website instructions may still allow Applebot to crawl your webpages. Your content will remain discoverable through Spotlight, Siri, and Safari, as well as other system-wide features on Apple devices.
 Traffic coming from Applebot is generally identified by using reverse DNS in the *.
 applebot.apple.com
 domain.
@@ -11,9 +16,11 @@
 Applebot IP CIDRs
 .
 The host command can be used to determine if an IP address is part of Applebot. These examples show the host command and its result:
-$ host 17-58-101-179.applebot.apple.com 17-58-101-179.applebot.apple.com has address 17.58.101.179.
+$ host 17-58-101-179.applebot.apple.com
+17-58-101-179.applebot.apple.com has address 17.58.101.179.
 The host command can also be used to verify that the DNS points to the same IP address:
-$ host 17.58.101.179 179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
+$ host 17.58.101.179
+179.101.58.17.in-addr.arpa domain name pointer 17-58-101-179.applebot.apple.com.
 A user agent helps webmasters identify crawler traffic, so that they can get accurate access log reports of crawler activity and control access to the site via robots.txt.
 Applebot powers several user agents, including Search and Podcasts.
 For search web crawling and rendering, Applebot uses the following format:
@@ -23,23 +30,36 @@
 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
 Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4.1 Mobile/15E148 Safari/604.1 (Applebot/0.1; +http://www.apple.com/go/applebot)
 Occasionally, Applebot will update the browser version that it advertises while remaining in the general format above.
-iTMS traffic may also come from
+iTMS traffic may come from
 applebot.apple.com
 hosts, and will be identified by the following user agent:
 User-Agent: iTMS
-The iTMS user agent does not follow robots.txt, as it is not a general search crawler. It only crawls URLs associated with registered content on Apple Podcasts.
+The iTMS user agent does not follow robots.txt, as it is not a general search crawler. The iTMS user agent only crawls URLs associated with registered content on Apple Podcasts.
 Applebot respects standard robots.txt directives in general search crawls that are targeted at Applebot. In this example, Applebot doesn't try to crawl documents that are under /private/ or /not-allowed/:
-User-agent: Applebot Allow: / Disallow: /private/ User-agent: * Disallow: /not-allowed/
-If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions.
+User-agent: Applebot
+Allow: /
+Disallow: /private/
+User-agent: *
+Disallow: /not-allowed/
+If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions. Applebot does not follow crawl-delay.
+Applebot is engineered for efficiency and will adjust to minimize the impact on site owners. For example, to avoid overloading site servers, Applebot’s crawl rate adjusts automatically when a site slows down or returns errors. Apple also caches crawled content to reduce unnecessary crawling. Identifying content that doesn't need to be crawled lowers infrastructure costs for site owners and makes the internet more efficient as a whole.
 Applebot may render the content of your website within a browser. If javascript, CSS, and other resources are blocked via robots.txt, it may not be able to render the content properly. This includes XHR, JS, and CSS that the page might require.
 In order for Applebot to index the best content for the page, make sure that everything needed for a user to render the page is available to Applebot. Alternatively, make sure that the website renders cleanly, even if all of the resources are not available. This is often referred to as graceful degradation.
 Applebot supports robots meta tags in HTML documents. To specify robots rules in meta tags, put the tags in the <head> section of the document:
-<html><head> <meta name="robots" content="noindex"/> ... </head> <body>...</body> </html>
+<html>
+<head>
+<meta name="robots" content="noindex"/>
+...
+</head>
+<body>...</body>
+</html>
 Applebot also supports the following directives:
 noindex:
 Applebot won't index this page, and it won't appear in Spotlight or Siri Suggestions.
 nosnippet:
-Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title.
+Applebot won't generate a description or web answer for the page. Any suggestions to visit this URL will only include the page's title. Apple will not use data tagged
+nosnippet
+as additional context and up-to-date content when AI models are used to generate output for display in Apple products and services.
 nofollow:
 Applebot won't follow any links on the page.
 none:
@@ -47,11 +67,32 @@
 all:
 Applebot provides the document for suggestions and snippets the contents so that a short description of the page can appear next to a representative image. Applebot may follow links on the page to provide more suggestions.
 To put multiple directives in a single meta tag, use a comma-separated list or multiple meta tags.
-<meta name="robots" content="nosnippet, noindex”> <meta name="robots" content=“noindex"> <meta name="robots" content=“nosnippet">
+<meta name="robots" content="nosnippet, noindex”>
+<meta name="robots" content=“noindex">
+<meta name="robots" content=“nosnippet">
+<meta name="applebot" content=“nosnippet">
+Applebot also supports indexing directives delivered via the X-Robots-Tag HTTP response header. This is useful for non-HTML resources (such as PDFs or images) where meta tags can’t be used, or when you want to apply directives at the server level.To apply a directive to Applebot:
+X-Robots-Tag: applebot: nosnippet
+Applebot supports the
+schema.org
+isAccessibleForFree
+property to identify pages that are behind a paywall, metered access, or subscription. Add the property to your page-level structured data in JSON-LD:
+<script type="application/ld+json">
+{
+"@context": "https://schema.org",
+"isAccessibleForFree": false
+}
+</script>
+Pages marked
+isAccessibleForFree: false
+are eligible to appear in search results, but Applebot will not use that content as additional context when AI models are used to generate output for display in Apple products and services.This signal applies at the page level. Section-level markup using
+hasPart
+is not supported. To opt out of having your content used to train Apple’s foundation models, use Applebot-Extended described in the next section.
 In addition to following all robots.txt rules and directives, Apple has a secondary user agent, Applebot-Extended, that gives web publishers additional controls over how their website content can be used by Apple.
-With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
+With Applebot-Extended, web publishers can choose to opt out of their website content being used to train Apple’s general purpose foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools.
 You can add a rule in robots.txt to disallow Applebot-Extended, as follows:
-User-agent: Applebot-Extended Disallow: /private/
+User-agent: Applebot-Extended
+Disallow: /private/
 Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.
 Allowing Applebot-Extended will help improve the capabilities and quality of Apple’s generative AI models over time.
 Apple Search may take the following factors into account when ranking web search results:
@@ -70,4 +111,4 @@
 Contact the vendor
 for additional information.
 Published Date:
-April 25, 2025
\ No newline at end of file
+June 08, 2026
\ No newline at end of file