]>

Melbourne Australia mnot@mnot.net https://mnot.net/

Sydney Australia mt@lowentropy.net

Internet-Draft Web sites often block archiving crawlers because they host time-sensitive information. This specification documents a robots.txt extension, "Archive-Embargo", that can be used to request that such crawlers delay publication of information. This document updates RFC 9309 to add two directives that support embargoes. About This Document Status information for this document may be found at . information can be found at . Source for this draft and an issue tracker can be found at .

Introduction Many Web publishers choose not allow their content to be publicly archived, by disallowing archiving crawlers in robots.txt . Archiving is widely seen as a public good, enhancing the value of the Internet both for current and future users. In some cases, publishers are amenable to archiving in the long term, but sensitive (for various reasons, including commercial considerations) to immediate republication of content in an archive. This document proposes updates to to address this concern by allowing publishers to state an embargo period for content on their sites. adds an "Archive-Embargo" rule that sets an embargo period. defines an "Embargo-Allow" rule that conditionally allows access to resources for crawlers that respect embargos.

Notational Conventions The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

The "Archive-Embargo" Rule This document adds a new rule that associates an embargo period with a group. Its value indicates the length of the embargo period, measured from the earliest of the HTTP response's Last-Modified value (when present and parseable; see ) and the time when the content was first observed. The following values (along with their associated embargo periods) are supported:

• "w" - 7 days
• "m" - 31 days
• "q" - 90 days

Any other value is unsupported and results in the rule being ignored. Note that the embargo period uses case-insensitive matching, so that "Q" and "q" both indicate 90 days. During an embargo period, a crawler MUST NOT allow covered content to be republished in an archive. Its existence MAY be indicated in an archive (e.g. by a "tombstone" entry that includes the URL, response header fields, and/or a cryptographic digest of the content) so long as the response body is not included. The rule ABNF pattern from is extended as shown in .

ABNF for Archive-Embargo line

The "Embargo-Allow" Rule Sites wishing to set archive embargoes without knowledge of whether a particular crawler supports this protocol extension need a way to predicate an 'allow' rule on support for it. The "Embargo-Allow" rule serves this function. Its semantics are identical to the "Allow" rule in , except that it is only applicable when the crawler supports and honours the "Archive-Embargo" rule. This includes having "Embargo-Allow" override an identical-length "Disallow". This rule has no effect on its own. If there is no valid value for "Archive-Embargo" in the group, the "Embargo-Allow" rule MUST be ignored. The rule ABNF pattern from is extended as shown in .

ABNF for Embargo-Allow line

Examples The following illustrates use of a group that allows the specified crawler to access content under the /news path, so long as it embargoes publication of that content for a calendar quarter. This depends on knowing that the specified crawler supports this specification. The example below illustrates a group that allows any crawler that supports and honours this specification to crawl resources under the /news path, so long as they embargo that content for one month.

IANA Considerations This document has no actions for IANA.

Security Considerations Embargoing is not a security mechanism; it relies upon crawlers to honour the embargo. This mechanism cannot be used to request that the existence of resources be hidden or that an embargo is in force; see .

Normative References Internet technical specifications often need to define a formal syntax. Over the years, a modified version of Backus-Naur Form (BNF), called Augmented BNF (ABNF), has been popular among many Internet specifications. The current specification documents ABNF. It balances compactness and simplicity with reasonable representational power. The differences between standard BNF and ABNF involve naming rules, repetition, alternatives, order-independence, and value ranges. This specification also supplies additional rule definitions and encoding for a core lexical analyzer of the type common to several Internet specifications. [STANDARDS-TRACK] This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching. The Hypertext Transfer Protocol (HTTP) is a stateless application-level protocol for distributed, collaborative, hypertext information systems. This document describes the overall architecture of HTTP, establishes common terminology, and defines aspects of the protocol that are shared by all versions. In this definition are core protocol elements, extensibility mechanisms, and the "http" and "https" Uniform Resource Identifier (URI) schemes. This document updates RFC 3864 and obsoletes RFCs 2818, 7231, 7232, 7233, 7235, 7538, 7615, 7694, and portions of 7230. In many standards track documents several words are used to signify the requirements in the specification. These words are often capitalized. This document defines these words as they should be interpreted in IETF documents. This document specifies an Internet Best Current Practices for the Internet Community, and requests discussion and suggestions for improvements. RFC 2119 specifies common key words that may be used in protocol specifications. This document aims to reduce the ambiguity by clarifying that only UPPERCASE usage of the key words have the defined special meanings.

Implementing Embargoes in Archives How archives provide access to their contents will constrain how they might implement embargoes. This section considers archives that provide access to content from individual resources and archives that provide bulk snapshots that contain content from many resources. Both types of archive need to have a way to store or reconstruct an embargo date along with each representation that it stores. An archive that provides access to single items of content is simple: access to content can be limited until its embargo date. Managing access to archives that provide bulk access to content is more complex. To ensure that embargoed content is not included in archives, a complete and inaccessible repository containing all archived content. Periodically, content that has passed its embargo date can be copied to accessible versions of the repository. For large archives, this sort of update process could be time-consuming, so updates might only be applied periodically. For that reason, content might not be available immediately after the embargo date passes. Techniques like sharding of archives or maintaining a sorted index of embargo dates could make the update process more efficient or reduce the time between the embargo date passing and the content being available.