| Internet-Draft | Archive-Embargo | August 2026 |
| Nottingham & Thomson | Expires 3 March 2027 | [Page] |
Web sites often block archiving crawlers because they host time-sensitive information. This specification documents a robots.txt extension, "Archive-Embargo", that can be used to request that such crawlers delay publication of information.¶
This document updates RFC 9309 to add two directives that support embargoes.¶
This note is to be removed before publishing as an RFC.¶
Status information for this document may be found at https://datatracker.ietf.org/doc/draft-nottingham-archive-embargo/.¶
information can be found at https://projects.mnot.net/I-D/.¶
Source for this draft and an issue tracker can be found at https://github.com/mnot/I-D/labels/embargo.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 3 March 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Many Web publishers choose not allow their content to be publicly archived, by disallowing archiving crawlers in robots.txt [ROBOTS]. Archiving is widely seen as a public good, enhancing the value of the Internet both for current and future users.¶
In some cases, publishers are amenable to archiving in the long term, but sensitive (for various reasons, including commercial considerations) to immediate republication of content in an archive.¶
This document proposes updates to [ROBOTS] to address this concern by allowing publishers to state an embargo period for content on their sites.¶
Section 2 adds an "Archive-Embargo" rule that sets an embargo period. Section 3 defines an "Embargo-Allow" rule that conditionally allows access to resources for crawlers that respect embargos.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
This document adds a new rule that associates an embargo period with a group.¶
Its value indicates the length of the embargo period, measured from the earliest of the HTTP response's Last-Modified value (when present and parseable; see Section 8.8.2 of [HTTP]) and the time when the content was first observed. The following values (along with their associated embargo periods) are supported:¶
Any other value is unsupported and results in the rule being ignored. Note that the embargo period uses case-insensitive matching, so that "Q" and "q" both indicate 90 days.¶
During an embargo period, a crawler MUST NOT allow covered content to be republished in an archive. Its existence MAY be indicated in an archive (e.g. by a "tombstone" entry that includes the URL, response header fields, and/or a cryptographic digest of the content) so long as the response body is not included.¶
The rule ABNF [ABNF] pattern from Section 2.2 of [ROBOTS] is extended as shown in Figure 1.¶
rule =/ embargo archive-embargo = *WS "archive-embargo" *WS ":" *WS embargo-period EOL embargo-period = "w" / "m" / "q"
Sites wishing to set archive embargoes without knowledge of whether a particular crawler supports this protocol extension need a way to predicate an 'allow' rule on support for it. The "Embargo-Allow" rule serves this function.¶
Its semantics are identical to the "Allow" rule in Section 2.2.2 of [ROBOTS], except that it is only applicable when the crawler supports and honours the "Archive-Embargo" rule. This includes having "Embargo-Allow" override an identical-length "Disallow".¶
This rule has no effect on its own. If there is no valid value for "Archive-Embargo" in the group, the "Embargo-Allow" rule MUST be ignored.¶
The rule ABNF pattern from Section 2.2 of [ROBOTS] is extended as shown in Figure 2.¶
rule =/ embargo-allow
embargo-allow = *WS "embargo-allow" *WS ":" *WS
(path-pattern / empty-pattern) EOL
The following illustrates use of a group that allows the specified crawler to access content under the /news path, so long as it embargoes publication of that content for a calendar quarter.¶
This depends on knowing that the specified crawler supports this specification.¶
User-Agent: ExampleBot Archive-Embargo: q Allow: /news¶
The example below illustrates a group that allows any crawler that supports and honours this specification to crawl resources under the /news path, so long as they embargo that content for one month.¶
User-Agent: * Archive-Embargo: m Embargo-Allow: /news Disallow: /¶
This document has no actions for IANA.¶
Embargoing is not a security mechanism; it relies upon crawlers to honour the embargo.¶
This mechanism cannot be used to request that the existence of resources be hidden or that an embargo is in force; see Section 2.¶
How archives provide access to their contents will constrain how they might implement embargoes. This section considers archives that provide access to content from individual resources and archives that provide bulk snapshots that contain content from many resources.¶
Both types of archive need to have a way to store or reconstruct an embargo date along with each representation that it stores.¶
An archive that provides access to single items of content is simple: access to content can be limited until its embargo date.¶
Managing access to archives that provide bulk access to content is more complex. To ensure that embargoed content is not included in archives, a complete and inaccessible repository containing all archived content. Periodically, content that has passed its embargo date can be copied to accessible versions of the repository.¶
For large archives, this sort of update process could be time-consuming, so updates might only be applied periodically. For that reason, content might not be available immediately after the embargo date passes. Techniques like sharding of archives or maintaining a sorted index of embargo dates could make the update process more efficient or reduce the time between the embargo date passing and the content being available.¶