Network Working Group R. Mitchell Internet-Draft BotCentral Intended status: Informational 1 September 2026 Expires: 5 March 2027 The BotCentral Card: An Owner-Proven Consent Record for Automated Web Clients draft-mitchell-botcentral-card-00 Abstract The Robots Exclusion Protocol (RFC 9309) lets a site say which automated clients may fetch its content. It cannot express purpose: fetching a page to answer a person is not the same act as copying it into a model training set or taking an action on the site. It also cannot prove who published the policy. This document defines the BotCentral Card, a JSON record, one per domain, that states which purposes the domain owner consents to ("retrieve", "train", and "act" as separate answers), backed by a proof of domain control placed either in a DNS TXT record or at the well-known URI "/.well-known/botcentral.txt". Cards are written to a registry by authenticated publishers and read by any client over HTTP or the Model Context Protocol. Clients never write cards. A card is permission to be found; it is not a ranking and not a training grant. This document also registers the "botcentral.txt" well-known URI. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 5 March 2027. Mitchell Expires 5 March 2027 [Page 1] Internet-Draft BotCentral Card September 2026 Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 1.1. Requirements Language . . . . . . . . . . . . . . . . . . 4 1.2. Terminology . . . . . . . . . . . . . . . . . . . . . . . 4 2. Roles and Trust Model . . . . . . . . . . . . . . . . . . . . 4 3. The Card . . . . . . . . . . . . . . . . . . . . . . . . . . 4 3.1. Required Members . . . . . . . . . . . . . . . . . . . . 4 3.2. Consent . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.3. Scores . . . . . . . . . . . . . . . . . . . . . . . . . 6 3.4. Extensibility . . . . . . . . . . . . . . . . . . . . . . 6 4. Proof of Control . . . . . . . . . . . . . . . . . . . . . . 6 4.1. Verification Token . . . . . . . . . . . . . . . . . . . 6 4.2. Method "well-known-file" . . . . . . . . . . . . . . . . 6 4.3. Method "dns-txt" . . . . . . . . . . . . . . . . . . . . 7 4.4. Precedence and States . . . . . . . . . . . . . . . . . . 7 4.5. What Is Not Proof . . . . . . . . . . . . . . . . . . . . 7 5. Freshness . . . . . . . . . . . . . . . . . . . . . . . . . . 7 6. Registry Interface . . . . . . . . . . . . . . . . . . . . . 8 6.1. Discovery . . . . . . . . . . . . . . . . . . . . . . . . 8 6.2. Read . . . . . . . . . . . . . . . . . . . . . . . . . . 8 6.3. Model Context Protocol . . . . . . . . . . . . . . . . . 8 6.4. Write . . . . . . . . . . . . . . . . . . . . . . . . . . 8 6.5. Compiled robots.txt . . . . . . . . . . . . . . . . . . . 9 7. Client Obligations . . . . . . . . . . . . . . . . . . . . . 9 8. Security Considerations . . . . . . . . . . . . . . . . . . . 9 8.1. Server-Side Request Forgery . . . . . . . . . . . . . . . 9 8.2. HTML Shells . . . . . . . . . . . . . . . . . . . . . . . 9 8.3. Shared Hosting . . . . . . . . . . . . . . . . . . . . . 10 8.4. Token Replay and Domain Transfer . . . . . . . . . . . . 10 8.5. Publisher Compromise . . . . . . . . . . . . . . . . . . 10 8.6. Consent Is a Claim, Not an Enforcement . . . . . . . . . 10 8.7. Score Gaming . . . . . . . . . . . . . . . . . . . . . . 10 8.8. Privacy . . . . . . . . . . . . . . . . . . . . . . . . . 10 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10 9.1. Well-Known URI Registration . . . . . . . . . . . . . . . 10 9.2. Well-Known URI Registration: botcentral-index.json . . . 11 Mitchell Expires 5 March 2027 [Page 2] Internet-Draft BotCentral Card September 2026 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 11 10.1. Normative References . . . . . . . . . . . . . . . . . . 11 10.2. Informative References . . . . . . . . . . . . . . . . . 11 Appendix A. Example Card . . . . . . . . . . . . . . . . . . . . 12 Appendix B. Example Proof Files . . . . . . . . . . . . . . . . 13 Appendix C. Relationship to Existing Mechanisms . . . . . . . . 14 Appendix D. Acknowledgements . . . . . . . . . . . . . . . . . . 14 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 14 1. Introduction Search engines and web sites share a discovery layer that has accumulated over two decades: the Robots Exclusion Protocol [RFC9309], sitemaps, IndexNow [INDEXNOW], and, in advertising, "ads.txt" [ADSTXT]. Each of these is a small, public, owner-placed file that a large population of clients agrees to read. Automated clients that fetch the web on behalf of AI assistants have no such layer. They have "robots.txt", which was designed for crawlers and answers one question: may this product token fetch this path? It cannot say "you may retrieve this page to answer a user, but you may not train on it, and you may not act on it". It cannot say who owns the site. And it cannot be queried across the web; it must be fetched one origin at a time. In the absence of a shared record, every operator of such clients has built a private one. Each ships its own product tokens, often several per vendor to distinguish training from retrieval from search, together with its own submission form and its own allowlist. Site owners face an unbounded number of portals. Client operators cannot keep an unbounded number of allowlists honest. This document defines a single record, the BotCentral Card, that answers those questions once. The record is built from mechanisms that already exist: "robots.txt" [RFC9309] for crawl permission, DNS TXT records and well-known URIs [RFC8615] for proof of control, the TDM Reservation Protocol [TDMREP] for training reservation, OAuth 2.0 client credentials [RFC6749] for publisher authentication, and HTTP entity tags [RFC9110] for caching. The card adds the one thing those mechanisms lacked: a queryable index of who opted in, for which purpose, with proof. The design deliberately follows the shape of "ads.txt" and of the Web PKI: a proof on the origin, a public registry, and clients that refuse unsigned claims. Mitchell Expires 5 March 2027 [Page 3] Internet-Draft BotCentral Card September 2026 1.1. Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. 1.2. Terminology Owner The party that controls a domain, demonstrated by the proof in Section 4. Publisher A party authorized by an owner to write cards to a registry. Publishers authenticate with OAuth 2.0 client credentials [RFC6749]. A publisher is analogous to a seller in "ads.txt". Registry A service that verifies proof, stores cards, and serves them. The reference registry is operated at https://botcentral.org (https://botcentral.org). Client Any program that reads cards: a crawler, an assistant, a Model Context Protocol [MCP] host, or a search engine. Card The JSON record defined in Section 3. 2. Roles and Trust Model Owners prove control of a domain. Publishers write cards on behalf of owners. Registries verify proof, apply defaults, compute derived fields, and serve cards. Clients read cards and honor them. A registry MUST NOT accept a card write from a client. Only publishers write. A card written by a publisher does not become proven by being written; it becomes proven only when the registry independently confirms the proof in Section 4. 3. The Card A card is a JSON object served with media type "application/json". A registry MUST publish the card's JSON Schema [JSONSCHEMA] at "/v1/ schema.json". The normative field list is in [BOTCENTRAL]; this section summarizes the members whose semantics matter for interoperability. 3.1. Required Members botcentral The card version, "1.0" or "1.1". domain The registrable domain in lower case. canonical The site's canonical origin. MUST use the "https" scheme. name, summary, topics Display name, one-paragraph description, and Mitchell Expires 5 March 2027 [Page 4] Internet-Draft BotCentral Card September 2026 free-form topic tags. allow_bots Boolean master switch. When false, the registry MUST serve "retrieve", "train", and "act" as false regardless of what was submitted. allow, deny Lists of "robots.txt" product tokens the owner explicitly allows or denies. pointers Absolute URIs of the site's own "robots.txt", sitemap, and "llms.txt". Pointers are informational and are not proof. pages Declared URLs, each with a "url" and a "rel" role, and optionally a registry-computed "score" (Section 3.3). updated An [RFC3339] timestamp of the last publisher write. verification The proof state (Section 4). consent The purpose consent block (Section 3.2). 3.2. Consent The "consent" member carries five independent answers. retrieve (default true) The client may fetch content now to answer a person or to index it for search. This includes retrieval- augmented generation and citation. train (default false) The client may copy the origin's content into a model training set. act (default false) The client may take actions on the origin: submit forms, book, purchase, invoke tools, or write. cite (default true) Attribution to the origin is expected when its content is used. tdm (default "reserved") The TDM Reservation Protocol [TDMREP] value the owner asserts: "open", "reserved", or "licensed". The following rules apply. * Absent members take their defaults. A registry MUST apply defaults before serving a card. * "retrieve" true does not imply "train" true. Retrieval is not training. * "act" true states the owner's policy only. It grants no authentication or authorization on the origin itself. * A registry MAY mirror "tdm" at "/.well-known/tdmrep.json" for the domain. If the origin serves its own "tdmrep.json" and the values conflict, the origin's own file takes precedence. Mitchell Expires 5 March 2027 [Page 5] Internet-Draft BotCentral Card September 2026 3.3. Scores Each declared page MAY carry a "score", an integer from 0 to 100 computed by the registry from proof, consent, freshness, crawl health, page role, and surface. It is a hint for crawl ordering. Publishers MUST NOT set it, registries MUST ignore a submitted value, and clients MUST NOT present it as a ranking of sites against one another or as a claim about content quality. 3.4. Extensibility Unknown members MUST be preserved by registries and ignored by clients. 4. Proof of Control A card is proven by exactly one of two methods. Nothing else counts. 4.1. Verification Token The registry issues the publisher a per-domain verification token: an opaque string carrying at least 128 bits of entropy, encoded in URL- safe characters, bound to the pair (registry, domain). 4.2. Method "well-known-file" The owner serves the token at the well-known URI: https:///.well-known/botcentral.txt All of the following MUST hold for the proof to pass: 1. The HTTP status is 200. 2. The response "Content-Type" is not "text/html". 3. The first 400 octets of the body, compared case-insensitively, do not contain "". Conditions 2 and 3 exist because single-page-application hosts routinely return status 200 with an HTML shell for every path. An HTML 200 is a failed proof, not a passed one. The file MAY contain multiple "botcentral-verify=" lines, one per registry, and MAY contain comment lines beginning with "#". Mitchell Expires 5 March 2027 [Page 6] Internet-Draft BotCentral Card September 2026 4.3. Method "dns-txt" The owner publishes a DNS TXT record at the apex of the domain whose data contains "botcentral-verify=". Multiple TXT records at the apex are normal; the registry MUST scan all of them. Registries SHOULD resolve through a validating resolver. The choice of resolver is outside the scope of this document. 4.4. Precedence and States A registry checks Section 4.2 first, then Section 4.3. The first success is recorded in "verification.method". If neither succeeds, "verification.method" is one of: pending A token has been issued and no check has completed. unverified A check completed and failed. "verification.note" states why: origin unreachable, "robots.txt" unreachable, token absent, HTML shell returned, or "robots.txt" contains "Disallow: /" for a product token the card lists in "allow". stale A prior proof has not been re-confirmed within the freshness window (Section 5). "verification.checked" carries the [RFC3339] time of the last check, or null. 4.5. What Is Not Proof Submission through a publisher, including a registry operator's own publisher, is not proof. A reachable origin is not proof. A sitemap or "llms.txt" is not proof. An HTML "meta" element is not proof, deliberately: on a shared host any tenant can emit one, whereas the two accepted methods require control of DNS or of the "/.well-known/" path, which shared tenants lack. A registry MUST NOT serve "verification.method" of "dns-txt" or "well-known-file" for any card that did not pass Section 4.3 or Section 4.2 respectively. 5. Freshness Proof decays. A registry re-checks proof periodically. The default window is 168 hours; an owner MAY set "crawl.revalidate_hours" to another value. fresh Proven, and the last check is within the window. stale Proven previously; the last check is older than the window. unproven "verification.method" is "pending" or "unverified". Mitchell Expires 5 March 2027 [Page 7] Internet-Draft BotCentral Card September 2026 Clients SHOULD treat "stale" cards as listed but of lower confidence, and SHOULD treat "unproven" cards as unlisted for any purpose that depends on owner consent. 6. Registry Interface A registry exposes the following, relative to its origin. Anonymous reads MUST be permitted, subject to rate limiting. 6.1. Discovery GET /.well-known/botcentral-index.json Returns an object with the card version, the registry name, and absolute URIs for the catalog search, the contract documentation, the OpenAPI description, the schema, the change feed, the snapshot, and the MCP endpoint, together with a "listed" array of {domain, name, href} entries. All other registry resources are discoverable from this document. 6.2. Read GET /v1/site/{domain} GET /v1/search?q=&topic=&verified=&consent=&capability= GET /v1/changes GET /v1/snapshot GET /v1/schema.json GET /v1/openapi.json "/v1/site/{domain}" returns one card and MUST support "ETag" and "If- None-Match" [RFC9110]. "/v1/changes" is a feed of (domain, updated, etag) tuples since a cursor. "/v1/snapshot" returns a hash over the whole catalog so that a client can ask "has anything changed?" in one request, in the spirit of IndexNow [INDEXNOW]. 6.3. Model Context Protocol A registry SHOULD expose an MCP [MCP] server at "POST /mcp", advertised at "/.well-known/mcp.json", offering at minimum the tools "search_catalog" and "get_listed_site" with semantics identical to the read interface. 6.4. Write POST /oauth/token PUT /v1/publisher/sites/{domain} POST /v1/publisher/sites/{domain}/reverify Mitchell Expires 5 March 2027 [Page 8] Internet-Draft BotCentral Card September 2026 Publishers obtain tokens with the OAuth 2.0 client credentials grant [RFC6749]; authorization server metadata is published per [RFC8414] and protected resource metadata per [RFC9728]. A write MUST NOT make a card fresh; only a passed proof check does that. 6.5. Compiled robots.txt A registry MAY compile a "robots.txt" [RFC9309] from a card for the owner's convenience. When it does, product tokens whose purpose is training receive "Disallow: /" unless "consent.train" is true; tokens whose purpose is retrieval remain allowed unless "consent.retrieve" is false; entries in "allow" and "deny" override. Some tokens, such as "Google-Extended" and "Applebot-Extended", are control tokens that never appear as HTTP User-Agent values; a registry that publishes an agent list MUST label them as such. 7. Client Obligations A client that reads a card and then fetches the origin: 1. MUST honor "consent" for its actual purpose. A retrieval fetch is permitted by "retrieve" true; using the result for training requires "train" true. 2. MUST still honor the origin's own "robots.txt". The card never overrides it. 3. SHOULD send "If-None-Match" on repeat card fetches and poll "/v1/ snapshot" rather than re-read the catalog. 4. MUST NOT treat "score" as a ranking or a quality claim. 5. SHOULD attribute the origin when "cite" is true. 8. Security Considerations 8.1. Server-Side Request Forgery A registry fetches owner-supplied URLs to verify proof. It MUST resolve the target hostname before connecting and reject private, loopback, link-local, and cloud metadata address ranges; MUST require the "https" scheme; MUST cap redirects; and MUST enforce a timeout and a response body size limit. 8.2. HTML Shells Single-page-application hosts return status 200 for every path. The HTML rejection rules in Section 4.2 are normative for this reason. A registry that accepted an HTML 200 as proof would let any tenant of such a host claim any path. Mitchell Expires 5 March 2027 [Page 9] Internet-Draft BotCentral Card September 2026 8.3. Shared Hosting Page content and "meta" elements are excluded from proof because any tenant of a shared host can emit them. The two accepted methods require control of DNS or of the "/.well-known/" path. 8.4. Token Replay and Domain Transfer A token proves control at check time only. The freshness window in Section 5 bounds how long a lapsed or transferred domain keeps a proven card. Registries SHOULD shorten the window for domains whose registration data changes. 8.5. Publisher Compromise A publisher can rewrite the cards it controls. Registries MUST record the writing publisher in "listed_by", MUST log writes, and SHOULD provide owners a means to revoke a publisher's authority. 8.6. Consent Is a Claim, Not an Enforcement A card cannot stop a client that ignores it. Its value is the same as that of "robots.txt" and "ads.txt": a public, provable record against which behavior can be audited. 8.7. Score Gaming "score" is derived from proof, consent, freshness, crawl health, page role, and surface, none of which a publisher sets directly. Registries MUST NOT let a publisher influence "score" except through those underlying facts. 8.8. Privacy Cards contain only information the owner chose to publish about a domain. A registry's change feed reveals when a domain's policy changed, which is intended. Registries SHOULD NOT log the identity of anonymous readers beyond what rate limiting requires. 9. IANA Considerations 9.1. Well-Known URI Registration IANA is requested to register the following in the "Well-Known URIs" registry established by [RFC8615]: URI suffix botcentral.txt Change controller Rodney Mitchell (rodneymitchellv@gmail.com) Mitchell Expires 5 March 2027 [Page 10] Internet-Draft BotCentral Card September 2026 Specification document This document, Section 4.2 Status permanent Related information The resource is plain text and MUST NOT be served with media type "text/html". See Section 4.2. 9.2. Well-Known URI Registration: botcentral-index.json IANA is requested to register the following in the same registry: URI suffix botcentral-index.json Change controller Rodney Mitchell (rodneymitchellv@gmail.com) Specification document This document, Section 6 Status permanent 10. References 10.1. Normative References [JSONSCHEMA] Wright, A., Andrews, H., Hutton, B., and G. Dennis, "JSON Schema: A Media Type for Describing JSON Documents", 2022, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . [RFC8414] Jones, M., Sakimura, N., and J. Bradley, "OAuth 2.0 Authorization Server Metadata", RFC 8414, DOI 10.17487/RFC8414, June 2018, . [RFC9728] Jones, M.B., Hunt, P., and A. Parecki, "OAuth 2.0 Protected Resource Metadata", RFC 9728, DOI 10.17487/RFC9728, April 2025, . 10.2. Informative References [ADSTXT] IAB Technology Laboratory, "ads.txt Specification Version 1.1", 2022, . Mitchell Expires 5 March 2027 [Page 11] Internet-Draft BotCentral Card September 2026 [BOTCENTRAL] Mitchell, R., "BotCentral Card Specification 1.1", 2026, . [INDEXNOW] Microsoft Bing and Yandex, "IndexNow Protocol", 2021, . [MCP] Anthropic, PBC, "Model Context Protocol Specification", 2025, . [RFC3339] Klyne, G. and C. Newman, "Date and Time on the Internet: Timestamps", RFC 3339, DOI 10.17487/RFC3339, July 2002, . [RFC6749] Hardt, D., Ed., "The OAuth 2.0 Authorization Framework", RFC 6749, DOI 10.17487/RFC6749, October 2012, . [RFC8615] Nottingham, M., "Well-Known Uniform Resource Identifiers (URIs)", RFC 8615, DOI 10.17487/RFC8615, May 2019, . [RFC9110] Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, . [RFC9309] Koster, M., Illyes, G., Zeller, H., and L. Sassman, "Robots Exclusion Protocol", RFC 9309, DOI 10.17487/RFC9309, September 2022, . [TDMREP] W3C TDM Reservation Protocol Community Group, "TDM Reservation Protocol (TDMRep)", 2024, . Appendix A. Example Card Mitchell Expires 5 March 2027 [Page 12] Internet-Draft BotCentral Card September 2026 { "botcentral": "1.1", "domain": "fieldnote.press", "canonical": "https://fieldnote.press/", "name": "Field Note", "summary": "Independent reporting on local government.", "topics": ["news", "infrastructure"], "allow_bots": true, "allow": ["ClaudeBot", "OAI-SearchBot"], "deny": [], "pointers": { "robots": "https://fieldnote.press/robots.txt", "sitemap": "https://fieldnote.press/sitemap.xml", "llms": "https://fieldnote.press/llms.txt" }, "pages": [ { "url": "https://fieldnote.press/", "rel": "home", "score": 94 } ], "updated": "2026-08-30T14:02:11Z", "verification": { "method": "well-known-file", "checked": "2026-08-31T09:10:00Z", "note": "Token matched /.well-known/botcentral.txt (plain text)." }, "consent": { "retrieve": true, "train": false, "act": false, "cite": true, "tdm": "reserved" }, "freshness": { "status": "fresh", "checked": "2026-08-31T09:10:00Z", "revalidate_hours": 168 }, "listed_by": { "id": "citefleet", "name": "CiteFleet" } } Appendix B. Example Proof Files A "/.well-known/botcentral.txt": # BotCentral proof of control. Plain text. Not HTML. botcentral-verify=Kx7mQ2vB9pR4tY8wZ1aC3dF6gH0jL5nP A DNS TXT record at the apex: Mitchell Expires 5 March 2027 [Page 13] Internet-Draft BotCentral Card September 2026 fieldnote.press. 3600 IN TXT "botcentral-verify=Kx7mQ2vB9pR4tY8wZ1aC3dF6gH0jL5nP" Appendix C. Relationship to Existing Mechanisms robots.txt [RFC9309] Answers "may this token fetch this path?". The card answers "for what purpose, and says who?". The card never overrides "robots.txt". TDM Reservation Protocol [TDMREP] Expresses a training reservation for an origin. The card carries the same value in "consent.tdm" and defers to the origin's own file on conflict. ads.txt [ADSTXT] The model for this design: an owner-placed plain- text file on the origin, authorized sellers, and buyers that refuse unlisted claims. IndexNow [INDEXNOW] The model for the change feed and snapshot: tell clients what changed instead of making them recrawl. Appendix D. Acknowledgements The structure of this document follows the guidance at authors.ietf.org. The "ads.txt" and IndexNow designs informed the trust model and the change feed respectively. Author's Address Rodney Mitchell BotCentral Email: rodneymitchellv@gmail.com URI: https://botcentral.org Mitchell Expires 5 March 2027 [Page 14]