BotCentral

rodneymitchellv@gmail.com https://botcentral.org

Applications robots.txt consent well-known AI crawlers MCP The Robots Exclusion Protocol (RFC 9309) lets a site say which automated clients may fetch its content. It cannot express purpose: fetching a page to answer a person is not the same act as copying it into a model training set or taking an action on the site. It also cannot prove who published the policy. This document defines the BotCentral Card, a JSON record, one per domain, that states which purposes the domain owner consents to ("retrieve", "train", and "act" as separate answers), backed by a proof of domain control placed either in a DNS TXT record or at the well-known URI "/.well-known/botcentral.txt". Cards are written to a registry by authenticated publishers and read by any client over HTTP or the Model Context Protocol. Clients never write cards. A card is permission to be found; it is not a ranking and not a training grant. This document also registers the "botcentral.txt" well-known URI.

Introduction Search engines and web sites share a discovery layer that has accumulated over two decades: the Robots Exclusion Protocol , sitemaps, IndexNow , and, in advertising, "ads.txt" . Each of these is a small, public, owner-placed file that a large population of clients agrees to read. Automated clients that fetch the web on behalf of AI assistants have no such layer. They have "robots.txt", which was designed for crawlers and answers one question: may this product token fetch this path? It cannot say "you may retrieve this page to answer a user, but you may not train on it, and you may not act on it". It cannot say who owns the site. And it cannot be queried across the web; it must be fetched one origin at a time. In the absence of a shared record, every operator of such clients has built a private one. Each ships its own product tokens, often several per vendor to distinguish training from retrieval from search, together with its own submission form and its own allowlist. Site owners face an unbounded number of portals. Client operators cannot keep an unbounded number of allowlists honest. This document defines a single record, the BotCentral Card, that answers those questions once. The record is built from mechanisms that already exist: "robots.txt" for crawl permission, DNS TXT records and well-known URIs for proof of control, the TDM Reservation Protocol for training reservation, OAuth 2.0 client credentials for publisher authentication, and HTTP entity tags for caching. The card adds the one thing those mechanisms lacked: a queryable index of who opted in, for which purpose, with proof. The design deliberately follows the shape of "ads.txt" and of the Web PKI: a proof on the origin, a public registry, and clients that refuse unsigned claims.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Terminology

Owner

The party that controls a domain, demonstrated by the proof in .

Publisher

A party authorized by an owner to write cards to a registry. Publishers authenticate with OAuth 2.0 client credentials . A publisher is analogous to a seller in "ads.txt".

Registry

A service that verifies proof, stores cards, and serves them. The reference registry is operated at https://botcentral.org.

Client

Any program that reads cards: a crawler, an assistant, a Model Context Protocol host, or a search engine.

Card

The JSON record defined in .

Roles and Trust Model Owners prove control of a domain. Publishers write cards on behalf of owners. Registries verify proof, apply defaults, compute derived fields, and serve cards. Clients read cards and honor them. A registry MUST NOT accept a card write from a client. Only publishers write. A card written by a publisher does not become proven by being written; it becomes proven only when the registry independently confirms the proof in .

The Card A card is a JSON object served with media type "application/json". A registry MUST publish the card's JSON Schema at "/v1/schema.json". The normative field list is in ; this section summarizes the members whose semantics matter for interoperability.

Required Members

botcentral

The card version, "1.0" or "1.1".

domain

The registrable domain in lower case.

canonical

The site's canonical origin. MUST use the "https" scheme.

name, summary, topics

Display name, one-paragraph description, and free-form topic tags.

allow_bots

Boolean master switch. When false, the registry MUST serve "retrieve", "train", and "act" as false regardless of what was submitted.

allow, deny

Lists of "robots.txt" product tokens the owner explicitly allows or denies.

pointers

Absolute URIs of the site's own "robots.txt", sitemap, and "llms.txt". Pointers are informational and are not proof.

pages

Declared URLs, each with a "url" and a "rel" role, and optionally a registry-computed "score" ().

updated

An timestamp of the last publisher write.

verification

The proof state ().

consent

The purpose consent block ().

Consent The "consent" member carries five independent answers.

retrieve (default true)

The client may fetch content now to answer a person or to index it for search. This includes retrieval-augmented generation and citation.

train (default false)

The client may copy the origin's content into a model training set.

act (default false)

The client may take actions on the origin: submit forms, book, purchase, invoke tools, or write.

cite (default true)

Attribution to the origin is expected when its content is used.

tdm (default "reserved")

The TDM Reservation Protocol value the owner asserts: "open", "reserved", or "licensed".

The following rules apply.

• Absent members take their defaults. A registry MUST apply defaults before serving a card.
• "retrieve" true does not imply "train" true. Retrieval is not training.
• "act" true states the owner's policy only. It grants no authentication or authorization on the origin itself.
• A registry MAY mirror "tdm" at "/.well-known/tdmrep.json" for the domain. If the origin serves its own "tdmrep.json" and the values conflict, the origin's own file takes precedence.

Scores Each declared page MAY carry a "score", an integer from 0 to 100 computed by the registry from proof, consent, freshness, crawl health, page role, and surface. It is a hint for crawl ordering. Publishers MUST NOT set it, registries MUST ignore a submitted value, and clients MUST NOT present it as a ranking of sites against one another or as a claim about content quality.

Extensibility Unknown members MUST be preserved by registries and ignored by clients.

Proof of Control A card is proven by exactly one of two methods. Nothing else counts.

Verification Token The registry issues the publisher a per-domain verification token: an opaque string carrying at least 128 bits of entropy, encoded in URL-safe characters, bound to the pair (registry, domain).

Method "well-known-file" The owner serves the token at the well-known URI: /.well-known/botcentral.txt ]]> All of the following MUST hold for the proof to pass:

1. The HTTP status is 200.
2. The response "Content-Type" is not "text/html".
3. The first 400 octets of the body, compared case-insensitively, do not contain "<!doctype", "<html", "<head", or "<body".
4. The body contains the line "botcentral-verify=<token>".

Conditions 2 and 3 exist because single-page-application hosts routinely return status 200 with an HTML shell for every path. An HTML 200 is a failed proof, not a passed one. The file MAY contain multiple "botcentral-verify=" lines, one per registry, and MAY contain comment lines beginning with "#".

Method "dns-txt" The owner publishes a DNS TXT record at the apex of the domain whose data contains "botcentral-verify=<token>". Multiple TXT records at the apex are normal; the registry MUST scan all of them. Registries SHOULD resolve through a validating resolver. The choice of resolver is outside the scope of this document.

Precedence and States A registry checks first, then . The first success is recorded in "verification.method". If neither succeeds, "verification.method" is one of:

pending

A token has been issued and no check has completed.

unverified

A check completed and failed. "verification.note" states why: origin unreachable, "robots.txt" unreachable, token absent, HTML shell returned, or "robots.txt" contains "Disallow: /" for a product token the card lists in "allow".

stale

A prior proof has not been re-confirmed within the freshness window ().

"verification.checked" carries the time of the last check, or null.

What Is Not Proof Submission through a publisher, including a registry operator's own publisher, is not proof. A reachable origin is not proof. A sitemap or "llms.txt" is not proof. An HTML "meta" element is not proof, deliberately: on a shared host any tenant can emit one, whereas the two accepted methods require control of DNS or of the "/.well-known/" path, which shared tenants lack. A registry MUST NOT serve "verification.method" of "dns-txt" or "well-known-file" for any card that did not pass or respectively.

Freshness Proof decays. A registry re-checks proof periodically. The default window is 168 hours; an owner MAY set "crawl.revalidate_hours" to another value.

fresh

Proven, and the last check is within the window.

stale

Proven previously; the last check is older than the window.

unproven

"verification.method" is "pending" or "unverified".

Clients SHOULD treat "stale" cards as listed but of lower confidence, and SHOULD treat "unproven" cards as unlisted for any purpose that depends on owner consent.

Registry Interface A registry exposes the following, relative to its origin. Anonymous reads MUST be permitted, subject to rate limiting.

Discovery Returns an object with the card version, the registry name, and absolute URIs for the catalog search, the contract documentation, the OpenAPI description, the schema, the change feed, the snapshot, and the MCP endpoint, together with a "listed" array of {domain, name, href} entries. All other registry resources are discoverable from this document.

Read "/v1/site/{domain}" returns one card and MUST support "ETag" and "If-None-Match" . "/v1/changes" is a feed of (domain, updated, etag) tuples since a cursor. "/v1/snapshot" returns a hash over the whole catalog so that a client can ask "has anything changed?" in one request, in the spirit of IndexNow .

Model Context Protocol A registry SHOULD expose an MCP server at "POST /mcp", advertised at "/.well-known/mcp.json", offering at minimum the tools "search_catalog" and "get_listed_site" with semantics identical to the read interface.

Write Publishers obtain tokens with the OAuth 2.0 client credentials grant ; authorization server metadata is published per and protected resource metadata per . A write MUST NOT make a card fresh; only a passed proof check does that.

Compiled robots.txt A registry MAY compile a "robots.txt" from a card for the owner's convenience. When it does, product tokens whose purpose is training receive "Disallow: /" unless "consent.train" is true; tokens whose purpose is retrieval remain allowed unless "consent.retrieve" is false; entries in "allow" and "deny" override. Some tokens, such as "Google-Extended" and "Applebot-Extended", are control tokens that never appear as HTTP User-Agent values; a registry that publishes an agent list MUST label them as such.

Client Obligations A client that reads a card and then fetches the origin:

1. MUST honor "consent" for its actual purpose. A retrieval fetch is permitted by "retrieve" true; using the result for training requires "train" true.
2. MUST still honor the origin's own "robots.txt". The card never overrides it.
3. SHOULD send "If-None-Match" on repeat card fetches and poll "/v1/snapshot" rather than re-read the catalog.
4. MUST NOT treat "score" as a ranking or a quality claim.
5. SHOULD attribute the origin when "cite" is true.

Security Considerations

Server-Side Request Forgery A registry fetches owner-supplied URLs to verify proof. It MUST resolve the target hostname before connecting and reject private, loopback, link-local, and cloud metadata address ranges; MUST require the "https" scheme; MUST cap redirects; and MUST enforce a timeout and a response body size limit.

HTML Shells Single-page-application hosts return status 200 for every path. The HTML rejection rules in are normative for this reason. A registry that accepted an HTML 200 as proof would let any tenant of such a host claim any path.

Shared Hosting Page content and "meta" elements are excluded from proof because any tenant of a shared host can emit them. The two accepted methods require control of DNS or of the "/.well-known/" path.

Token Replay and Domain Transfer A token proves control at check time only. The freshness window in bounds how long a lapsed or transferred domain keeps a proven card. Registries SHOULD shorten the window for domains whose registration data changes.

Publisher Compromise A publisher can rewrite the cards it controls. Registries MUST record the writing publisher in "listed_by", MUST log writes, and SHOULD provide owners a means to revoke a publisher's authority.

Consent Is a Claim, Not an Enforcement A card cannot stop a client that ignores it. Its value is the same as that of "robots.txt" and "ads.txt": a public, provable record against which behavior can be audited.

Score Gaming "score" is derived from proof, consent, freshness, crawl health, page role, and surface, none of which a publisher sets directly. Registries MUST NOT let a publisher influence "score" except through those underlying facts.

Privacy Cards contain only information the owner chose to publish about a domain. A registry's change feed reveals when a domain's policy changed, which is intended. Registries SHOULD NOT log the identity of anonymous readers beyond what rate limiting requires.

IANA Considerations

Well-Known URI Registration IANA is requested to register the following in the "Well-Known URIs" registry established by :

URI suffix

botcentral.txt

Change controller

Rodney Mitchell (rodneymitchellv@gmail.com)

Specification document

This document,

Status

permanent

Related information

The resource is plain text and MUST NOT be served with media type "text/html". See .

Well-Known URI Registration: botcentral-index.json IANA is requested to register the following in the same registry:

URI suffix

botcentral-index.json

Change controller

Rodney Mitchell (rodneymitchellv@gmail.com)

Specification document

This document,

Status

permanent

References Normative References Informative References IAB Technology Laboratory Microsoft Bing and Yandex Anthropic, PBC W3C TDM Reservation Protocol Community Group

Example Card

Example Proof Files A "/.well-known/botcentral.txt": A DNS TXT record at the apex:

Relationship to Existing Mechanisms

robots.txt

Answers "may this token fetch this path?". The card answers "for what purpose, and says who?". The card never overrides "robots.txt".

TDM Reservation Protocol

Expresses a training reservation for an origin. The card carries the same value in "consent.tdm" and defers to the origin's own file on conflict.

ads.txt

The model for this design: an owner-placed plain-text file on the origin, authorized sellers, and buyers that refuse unlisted claims.

IndexNow

The model for the change feed and snapshot: tell clients what changed instead of making them recrawl.

Acknowledgements The structure of this document follows the guidance at authors.ietf.org. The "ads.txt" and IndexNow designs informed the trust model and the change feed respectively.