| Internet-Draft | BotCentral Card | September 2026 |
| Mitchell | Expires 5 March 2027 | [Page] |
The Robots Exclusion Protocol (RFC 9309) lets a site say which automated clients may fetch its content. It cannot express purpose: fetching a page to answer a person is not the same act as copying it into a model training set or taking an action on the site. It also cannot prove who published the policy.¶
This document defines the BotCentral Card, a JSON record, one per domain, that states which purposes the domain owner consents to ("retrieve", "train", and "act" as separate answers), backed by a proof of domain control placed either in a DNS TXT record or at the well-known URI "/.well-known/botcentral.txt". Cards are written to a registry by authenticated publishers and read by any client over HTTP or the Model Context Protocol. Clients never write cards. A card is permission to be found; it is not a ranking and not a training grant.¶
This document also registers the "botcentral.txt" well-known URI.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 5 March 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document.¶
Search engines and web sites share a discovery layer that has accumulated over two decades: the Robots Exclusion Protocol [RFC9309], sitemaps, IndexNow [INDEXNOW], and, in advertising, "ads.txt" [ADSTXT]. Each of these is a small, public, owner-placed file that a large population of clients agrees to read.¶
Automated clients that fetch the web on behalf of AI assistants have no such layer. They have "robots.txt", which was designed for crawlers and answers one question: may this product token fetch this path? It cannot say "you may retrieve this page to answer a user, but you may not train on it, and you may not act on it". It cannot say who owns the site. And it cannot be queried across the web; it must be fetched one origin at a time.¶
In the absence of a shared record, every operator of such clients has built a private one. Each ships its own product tokens, often several per vendor to distinguish training from retrieval from search, together with its own submission form and its own allowlist. Site owners face an unbounded number of portals. Client operators cannot keep an unbounded number of allowlists honest.¶
This document defines a single record, the BotCentral Card, that answers those questions once. The record is built from mechanisms that already exist: "robots.txt" [RFC9309] for crawl permission, DNS TXT records and well-known URIs [RFC8615] for proof of control, the TDM Reservation Protocol [TDMREP] for training reservation, OAuth 2.0 client credentials [RFC6749] for publisher authentication, and HTTP entity tags [RFC9110] for caching. The card adds the one thing those mechanisms lacked: a queryable index of who opted in, for which purpose, with proof.¶
The design deliberately follows the shape of "ads.txt" and of the Web PKI: a proof on the origin, a public registry, and clients that refuse unsigned claims.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
Owners prove control of a domain. Publishers write cards on behalf of owners. Registries verify proof, apply defaults, compute derived fields, and serve cards. Clients read cards and honor them.¶
A registry MUST NOT accept a card write from a client. Only publishers write. A card written by a publisher does not become proven by being written; it becomes proven only when the registry independently confirms the proof in Section 4.¶
A card is a JSON object served with media type "application/json". A registry MUST publish the card's JSON Schema [JSONSCHEMA] at "/v1/schema.json". The normative field list is in [BOTCENTRAL]; this section summarizes the members whose semantics matter for interoperability.¶
The "consent" member carries five independent answers.¶
The following rules apply.¶
Each declared page MAY carry a "score", an integer from 0 to 100 computed by the registry from proof, consent, freshness, crawl health, page role, and surface. It is a hint for crawl ordering. Publishers MUST NOT set it, registries MUST ignore a submitted value, and clients MUST NOT present it as a ranking of sites against one another or as a claim about content quality.¶
Unknown members MUST be preserved by registries and ignored by clients.¶
A card is proven by exactly one of two methods. Nothing else counts.¶
The registry issues the publisher a per-domain verification token: an opaque string carrying at least 128 bits of entropy, encoded in URL-safe characters, bound to the pair (registry, domain).¶
The owner serves the token at the well-known URI:¶
https://<domain>/.well-known/botcentral.txt¶
All of the following MUST hold for the proof to pass:¶
Conditions 2 and 3 exist because single-page-application hosts routinely return status 200 with an HTML shell for every path. An HTML 200 is a failed proof, not a passed one.¶
The file MAY contain multiple "botcentral-verify=" lines, one per registry, and MAY contain comment lines beginning with "#".¶
The owner publishes a DNS TXT record at the apex of the domain whose data contains "botcentral-verify=<token>". Multiple TXT records at the apex are normal; the registry MUST scan all of them. Registries SHOULD resolve through a validating resolver. The choice of resolver is outside the scope of this document.¶
A registry checks Section 4.2 first, then Section 4.3. The first success is recorded in "verification.method". If neither succeeds, "verification.method" is one of:¶
"verification.checked" carries the [RFC3339] time of the last check, or null.¶
Submission through a publisher, including a registry operator's own publisher, is not proof. A reachable origin is not proof. A sitemap or "llms.txt" is not proof. An HTML "meta" element is not proof, deliberately: on a shared host any tenant can emit one, whereas the two accepted methods require control of DNS or of the "/.well-known/" path, which shared tenants lack.¶
A registry MUST NOT serve "verification.method" of "dns-txt" or "well-known-file" for any card that did not pass Section 4.3 or Section 4.2 respectively.¶
Proof decays. A registry re-checks proof periodically. The default window is 168 hours; an owner MAY set "crawl.revalidate_hours" to another value.¶
Clients SHOULD treat "stale" cards as listed but of lower confidence, and SHOULD treat "unproven" cards as unlisted for any purpose that depends on owner consent.¶
A registry exposes the following, relative to its origin. Anonymous reads MUST be permitted, subject to rate limiting.¶
GET /.well-known/botcentral-index.json¶
Returns an object with the card version, the registry name, and absolute URIs for the catalog search, the contract documentation, the OpenAPI description, the schema, the change feed, the snapshot, and the MCP endpoint, together with a "listed" array of {domain, name, href} entries. All other registry resources are discoverable from this document.¶
GET /v1/site/{domain}
GET /v1/search?q=&topic=&verified=&consent=&capability=
GET /v1/changes
GET /v1/snapshot
GET /v1/schema.json
GET /v1/openapi.json
¶
"/v1/site/{domain}" returns one card and MUST support "ETag" and "If-None-Match" [RFC9110]. "/v1/changes" is a feed of (domain, updated, etag) tuples since a cursor. "/v1/snapshot" returns a hash over the whole catalog so that a client can ask "has anything changed?" in one request, in the spirit of IndexNow [INDEXNOW].¶
A registry SHOULD expose an MCP [MCP] server at "POST /mcp", advertised at "/.well-known/mcp.json", offering at minimum the tools "search_catalog" and "get_listed_site" with semantics identical to the read interface.¶
POST /oauth/token
PUT /v1/publisher/sites/{domain}
POST /v1/publisher/sites/{domain}/reverify
¶
Publishers obtain tokens with the OAuth 2.0 client credentials grant [RFC6749]; authorization server metadata is published per [RFC8414] and protected resource metadata per [RFC9728]. A write MUST NOT make a card fresh; only a passed proof check does that.¶
A registry MAY compile a "robots.txt" [RFC9309] from a card for the owner's convenience. When it does, product tokens whose purpose is training receive "Disallow: /" unless "consent.train" is true; tokens whose purpose is retrieval remain allowed unless "consent.retrieve" is false; entries in "allow" and "deny" override. Some tokens, such as "Google-Extended" and "Applebot-Extended", are control tokens that never appear as HTTP User-Agent values; a registry that publishes an agent list MUST label them as such.¶
A client that reads a card and then fetches the origin:¶
A registry fetches owner-supplied URLs to verify proof. It MUST resolve the target hostname before connecting and reject private, loopback, link-local, and cloud metadata address ranges; MUST require the "https" scheme; MUST cap redirects; and MUST enforce a timeout and a response body size limit.¶
Single-page-application hosts return status 200 for every path. The HTML rejection rules in Section 4.2 are normative for this reason. A registry that accepted an HTML 200 as proof would let any tenant of such a host claim any path.¶
A token proves control at check time only. The freshness window in Section 5 bounds how long a lapsed or transferred domain keeps a proven card. Registries SHOULD shorten the window for domains whose registration data changes.¶
A publisher can rewrite the cards it controls. Registries MUST record the writing publisher in "listed_by", MUST log writes, and SHOULD provide owners a means to revoke a publisher's authority.¶
A card cannot stop a client that ignores it. Its value is the same as that of "robots.txt" and "ads.txt": a public, provable record against which behavior can be audited.¶
"score" is derived from proof, consent, freshness, crawl health, page role, and surface, none of which a publisher sets directly. Registries MUST NOT let a publisher influence "score" except through those underlying facts.¶
IANA is requested to register the following in the "Well-Known URIs" registry established by [RFC8615]:¶
{
"botcentral": "1.1",
"domain": "fieldnote.press",
"canonical": "https://fieldnote.press/",
"name": "Field Note",
"summary": "Independent reporting on local government.",
"topics": ["news", "infrastructure"],
"allow_bots": true,
"allow": ["ClaudeBot", "OAI-SearchBot"],
"deny": [],
"pointers": {
"robots": "https://fieldnote.press/robots.txt",
"sitemap": "https://fieldnote.press/sitemap.xml",
"llms": "https://fieldnote.press/llms.txt"
},
"pages": [
{ "url": "https://fieldnote.press/", "rel": "home", "score": 94 }
],
"updated": "2026-08-30T14:02:11Z",
"verification": {
"method": "well-known-file",
"checked": "2026-08-31T09:10:00Z",
"note": "Token matched /.well-known/botcentral.txt (plain text)."
},
"consent": {
"retrieve": true,
"train": false,
"act": false,
"cite": true,
"tdm": "reserved"
},
"freshness": {
"status": "fresh",
"checked": "2026-08-31T09:10:00Z",
"revalidate_hours": 168
},
"listed_by": { "id": "citefleet", "name": "CiteFleet" }
}
¶
A "/.well-known/botcentral.txt":¶
# BotCentral proof of control. Plain text. Not HTML. botcentral-verify=Kx7mQ2vB9pR4tY8wZ1aC3dF6gH0jL5nP¶
A DNS TXT record at the apex:¶
fieldnote.press. 3600 IN TXT
"botcentral-verify=Kx7mQ2vB9pR4tY8wZ1aC3dF6gH0jL5nP"
¶
The structure of this document follows the guidance at authors.ietf.org. The "ads.txt" and IndexNow designs informed the trust model and the change feed respectively.¶