A Standard Wire Format for Large Language Model Inference Streaming

1.1. Problem Statement

The AI inference ecosystem is experiencing protocol fragmentation reminiscent of the early web. Every major vendor has independently defined its own wire format for the same fundamental operation: streaming generated tokens from a model to a client. Each vendor uses a different combination of application-layer framing and JSON payload schema:¶

• OpenAI streams SSE with JSON payloads using the path choices[0].delta.content for generated text.¶

• Anthropic streams SSE with typed event blocks, placing generated text at delta.text inside content_block_delta events.¶

• Google Gemini streams SSE with JSON payloads using candidates[0].content.parts[0].text.¶

• Microsoft Azure / Copilot uses WebSocket with the SignalR Hub Protocol's [SignalR] proprietary 0x1E record-separator framing.¶

• Cursor and Windsurf (AI coding assistants) use the Connect protocol [Connect] (application/connect+proto) with a 5-byte binary envelope and Protobuf-encoded payloads over HTTP/2.¶

The underlying transports, specifically HTTP/2 [RFC9113], HTTP/1.1 [RFC9110], WebSocket [RFC6455], and SSE [WHATWG-HTML], are all properly standardized. The problem exists entirely at the application layer: the JSON schemas, event types, stream lifecycle conventions, and payload structures that ride on top of these transports.¶

Additionally, even where vendors use the same sub-protocol, behavior diverges in practice. For example, Cursor keeps HTTP/2 streams open indefinitely without sending END_STREAM, while Windsurf sends END_STREAM on the same Connect+proto protocol: same framing, different stream lifecycle.¶

SSE is used correctly per the W3C specification by most vendors, but nothing in the SSE specification constrains what goes inside the event: and data: fields. Event type names, JSON field names, nesting structures, and payload schemas are entirely up to each provider.¶

1.2. Use Cases

The absence of a standard wire format for AI inference streaming creates unnecessary cost and complexity across the AI ecosystem. The following use cases illustrate the breadth of the problem.¶

1.3. Scope

This document addresses Client/Application to LLM inference endpoint streaming, specifically the path from an end-user application (web browser, IDE, mobile app, API client) to an LLM inference endpoint and back.¶

The following are explicitly out of scope:¶

• Agent-to-tool interaction (e.g., Model Context Protocol [MCP])¶

• Agent-to-agent communication¶

• Model training traffic¶

• Embedding and batch inference APIs (non-streaming)¶

1.4. Design Rationale

The standard is designed around three principles:¶

1. Keep SSE as the transport. SSE over HTTP is already the dominant transport for AI inference streaming, used by OpenAI, Anthropic, Google, Cohere, and Mistral. It works on both HTTP/2 and HTTP/1.1, has broad ecosystem support, and requires no new transport protocol. A WebTransport [W3C-WEBTRANSPORT] binding MAY be defined in a future document as adoption matures.¶

2. Standardize only the payload. The transport layer is already standardized. This document defines only what goes inside the SSE event: and data: fields, specifically the event type taxonomy and JSON envelope schema.¶

3. Preserve vendor extensibility. Vendors retain full freedom to include proprietary metadata via namespaced extension fields. Conforming parsers MUST ignore unknown fields.¶

4.1. Request Media Type

This document registers the media type application/llm-request+json.¶

Sent by the client in the HTTP POST body, this Content-Type indicates that the request payload conforms to the request envelope defined in Section 4.2. Any component that observes this Content-Type can immediately identify the traffic as LLM inference, enabling automated routing, logging, and processing without hostname-based heuristics.¶

4.2. Request Envelope

The request body is a JSON object. Its top-level fields are organized into five logical groups: protocol control, conversation history, tool declarations, generation parameters, and extensibility.¶

4.3. Messages (Conversation History)

The messages field is an ordered array of message objects representing the conversation history. The model reads this full history to understand context before generating a response.¶

The role field identifies who produced the message:¶

"system":

Instructions that configure the model's behavior for the conversation. Typically the first message. Not visible to end users.¶

"user":

Input from the end user: the prompt, question, or instruction the model should respond to.¶

"assistant":

The model's own prior responses. Included in multi-turn conversations so the model has context of what it previously generated.¶

"tool":

A result returned from a tool invocation. When the model requests a tool call (via a tool.call event in the response), the client executes the tool locally and sends the result back as a "tool" message in the next request. The tool_call_id field MUST reference the id from the corresponding tool.call event.¶

A typical multi-turn conversation:¶

"messages": [
  {"role": "system",
   "content": "You are a coding assistant."},
  {"role": "user",
   "content": "Write a Python function to sort a list."},
  {"role": "assistant",
   "content": "def sort_list(items):\n    return sorted(items)"},
  {"role": "user",
   "content": "Now make it sort in reverse order."}
]

{"role": "user", "content": "What is the capital of France?"}

{"role": "user", "content": [
  {"type": "text", "text": "What is shown in this image?"},
  {"type": "image", "media_type": "image/png",
   "data": "<base64-encoded image>"}
]}

4.4. Tools (Function Declarations)

The tools field is an optional array of tool definitions. These are NOT part of the conversation; they are declarations of capabilities that the client can execute locally if the model decides to invoke them.¶

The model reads these definitions and, when it determines that a tool would help answer the user's request, emits a tool.call event in the response stream instead of generating text. The model itself does not execute anything; it outputs a structured request asking the client to run the specified function with the given arguments.¶

"tools": [
  {"name": "get_weather",
   "description": "Get current weather for a city.",
   "parameters": {
     "type": "object",
     "properties": {
       "city": {"type": "string",
                "description": "City name, e.g. Vienna"},
       "units": {"type": "string",
                 "enum": ["celsius", "fahrenheit"]}
     },
     "required": ["city"]
   }}
]

The complete tool interaction flow is described in Section 4.8.¶

4.5. Generation Parameters

The parameters field contains knobs that control how the model generates its response.¶

All fields within parameters are OPTIONAL. Vendors MAY define additional parameters. Unknown parameters MUST be ignored.¶

4.6. Request Metadata

The metadata field carries request-level information for tracing, billing, and audit. This data is NOT sent to the model and does NOT influence generation.¶

"metadata": {
  "request_id": "550e8400-e29b-41d4-a716-446655440000",
  "user_id": "user-12345",
  "session_id": "sess-67890"
}

The request_id field, if present, SHOULD be echoed in the stream.start and stream.end response events.¶

4.7. Vendor Extensions

The extensions field is an escape hatch for vendor-specific features. Keys MUST be namespaced using the format "vendor:name" to prevent collisions.¶

"extensions": {
  "vendor:openai": {"logprobs": true},
  "vendor:anthropic": {
    "thinking": {"type": "enabled",
                 "budget_tokens": 10000}}
}

Conforming implementations MUST ignore extension namespaces they do not recognize.¶

4.8. Tool Interaction Model

The model is stateless and cannot execute tools itself; it can only emit a structured request for the client to execute a tool on its behalf.¶

1. The client sends a request with messages and tools.¶

2. The model emits a tool.call event with the tool name and arguments, followed by content.stop with stop_reason "tool_use".¶

3. The client executes the named function locally.¶

4. The client sends a new request with the original history, the assistant's tool call, and a "tool" role message with the result.¶

5. The model reads the tool result and generates its final text response.¶

Request 1:
{"messages": [{"role": "user",
  "content": "What is the weather in Vienna?"}],
 "tools": [{"name": "get_weather", ...}]}

Response 1:
event: tool.call
data: {"type":"tool.call","id":"call_1",
  "name":"get_weather",
  "arguments":"{\"city\":\"Vienna\"}"}
event: content.stop
data: {"type":"content.stop",
  "stop_reason":"tool_use"}

Client executes get_weather("Vienna"), gets result.

Request 2:
{"messages": [
  {"role": "user",
   "content": "What is the weather in Vienna?"},
  {"role": "assistant", "content": null,
   "tool_calls": [{"id": "call_1",
     "name": "get_weather",
     "arguments": "{\"city\":\"Vienna\"}"}]},
  {"role": "tool", "tool_call_id": "call_1",
   "content": "Vienna: 24C, sunny"}],
 "tools": [{"name": "get_weather", ...}]}

Response 2:
event: content.delta
data: {"type":"content.delta",
  "delta":{"text":"The weather in Vienna is 24C."}}

The tool definitions travel with every request because the model is stateless.¶

4.9. Response Media Type

The SSE response uses the standard text/event-stream media type. Conformance is signaled by the combination of application/llm-request+json on the request and a stream.start event as the first event in the response.¶

5.1. SSE Event Framing

Each event on the wire consists of field lines followed by a blank line:¶

event: <event-type>\n
data: <JSON event envelope>\n
\n

The blank line (\n\n) is the event boundary. The HTTP transport may split bytes arbitrarily across frames or chunks; the SSE layer reassembles them.¶

HTTP layer:  Delivers raw bytes (DATA frames / chunks)
                  |
SSE layer:   Reassembles lines, uses \n\n as boundary
                  |
This spec:   Parses JSON envelope from data: field

The following constraints MUST be observed:¶

• Each SSE event MUST contain exactly one complete, valid JSON object in its data: field.¶

• A JSON event envelope MUST NOT span multiple SSE events.¶

• An endpoint MUST NOT emit partial JSON that requires a subsequent SSE event to complete.¶

5.2. Event Envelope

Every event carries a JSON object with three common fields:¶

The envelope follows a TLV-style structure:¶

T (Type):

The SSE event: field. Identifies the event type without parsing JSON.¶

L (Length):

Implicit. The SSE \n\n boundary defines the extent of the JSON body. See Section 6.¶

V (Value):

The JSON event envelope in the data: field.¶

Conforming parsers MUST ignore unknown fields (forward compatibility).¶

5.3. Event Types

event: stream.start
data: {"v":"1.0","seq":0,"type":"stream.start",
  "request_id":"550e8400-...",
  "model":"example-model","created":1749292800}

event: content.delta
data: {"v":"1.0","seq":1,"type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":"The weather in Vienna"}}

event: content.delta
data: {"v":"1.0","seq":2,"type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":" is currently 24C"}}

event: content.delta
data: {"v":"1.0","seq":3,"type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":" and sunny."}}

5.4. Stream Lifecycle

1. The stream begins with exactly one stream.start (seq 0).¶

2. Zero or more content.delta events deliver text incrementally.¶

3. Zero or more tool.call events request tool invocations.¶

4. Zero or more usage events report token consumption.¶

5. Each content block is terminated by content.stop.¶

6. If an error occurs, an error event is emitted.¶

7. The stream ends with exactly one stream.end.¶

       POST request
            |
      stream.start (seq=0)
            |
       STREAMING
      /    |    \
content.delta tool.call usage
      \    |    /
    content.stop | error
            |
       stream.end
            |
         CLOSED

5.5. Complete Example

HTTP/2 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

event: stream.start
data: {"v":"1.0","seq":0,
  "type":"stream.start",
  "request_id":"550e8400-...",
  "model":"example-model",
  "created":1749292800}

event: content.delta
data: {"v":"1.0","seq":1,
  "type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":"The"}}

event: content.delta
data: {"v":"1.0","seq":2,
  "type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":" weather"}}

event: content.delta
data: {"v":"1.0","seq":3,
  "type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":" in Vienna"}}

event: content.delta
data: {"v":"1.0","seq":4,
  "type":"content.delta",
  "index":0,"role":"assistant",
  "delta":{"text":" is 24C and sunny."}}

event: content.stop
data: {"v":"1.0","seq":5,
  "type":"content.stop",
  "index":0,"stop_reason":"end_turn"}

event: usage
data: {"v":"1.0","seq":6,
  "type":"usage",
  "input_tokens":42,"output_tokens":8}

event: stream.end
data: {"v":"1.0","seq":7,
  "type":"stream.end",
  "request_id":"550e8400-..."}

6.1. SSE Text-Based Framing (This Specification)

SSE delimits events with \n\n. The parser must scan the byte stream to locate each boundary. This is O(n) per event.¶

6.2. Length-Prefixed Binary Framing (Alternative)

A binary approach (e.g., gRPC's [gRPC] 5-byte envelope) prefixes each message with a type byte and 4-byte length. The parser reads the header and jumps directly to the next message. This is O(1) per boundary.¶

6.3. Rationale for SSE

1. Compatibility: SSE is supported natively by browser EventSource and every major HTTP library. Binary framing would require custom parsers.¶

2. Existing deployment: The five major SSE-based LLM providers would need minimal changes to adopt a standard JSON schema. Binary framing would require re-engineering their streaming pipeline.¶

3. Debuggability: SSE events are human-readable in browser developer tools, curl, and log files.¶

4. Incremental adoption: The payload schema is transport-independent. The same event types and JSON envelope can be carried over SSE or a future binary binding without changes.¶

A future companion document MAY define a binary transport binding, for example over WebTransport [W3C-WEBTRANSPORT], that provides length-prefixed framing while reusing the same event types and JSON envelope.¶

7.1. API Gateway Translation

An API gateway can accept vendor-specific requests, forward them to the upstream provider in their native format, and translate the response stream into the standard format before delivering it to the client.¶

7.2. Content-Type Negotiation

If a client sends Content-Type: application/llm-request+json and the endpoint does not support the standard, it SHOULD return 415 Unsupported Media Type, allowing the client to fall back.¶

7.3. Dual-Format Endpoints

A vendor MAY support both formats, selected by the request Content-Type or an Accept header:¶

Accept: text/event-stream; profile="llm-stream-1.0"

8.1. Prompt Confidentiality

Prompts in application/llm-request+json requests may contain highly sensitive data. Any system that logs or processes these requests MUST apply appropriate confidentiality protections.¶

8.2. Tool-Call Abuse

A compromised or adversarial endpoint could emit tool.call events invoking sensitive tools. Client implementations SHOULD validate tool.call events against the set of tools declared in the original request and SHOULD require user confirmation before executing tool invocations with side effects.¶

8.3. Extension Field Safety

The extensions field allows vendor-specific data. Implementations SHOULD NOT interpret or execute extension values from untrusted sources.¶

8.4. Sequence Exhaustion

The seq field is an integer. Implementations SHOULD use 64-bit integers. A stream exceeding 2^53 events (the safe integer limit in JSON/JavaScript) is pathological and SHOULD be terminated.¶

9.1. Media Type Registration: application/llm-request+json

Type name:

application¶

Subtype name:

llm-request+json¶

Required parameters:

None¶

Optional parameters:

version (default: "1.0")¶

Encoding considerations:

8bit (UTF-8 JSON)¶

Intended usage:

COMMON¶

9.2. SSE Event Type Registry

This document requests creation of an IANA registry for LLM-Stream SSE event types:¶

New event types are registered via Specification Required [RFC8126].¶

10.1. Normative References

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC6455]

Fette, I. and A. Melnikov, "The WebSocket Protocol", RFC 6455, DOI 10.17487/RFC6455, December 2011, <https://www.rfc-editor.org/rfc/rfc6455>.

[RFC8126]

Cotton, M., Leiba, B., and T. Narten, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 8126, DOI 10.17487/RFC8126, June 2017, <https://www.rfc-editor.org/rfc/rfc8126>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

[RFC8446]

Rescorla, E., "The Transport Layer Security (TLS) Protocol Version 1.3", RFC 8446, DOI 10.17487/RFC8446, August 2018, <https://www.rfc-editor.org/rfc/rfc8446>.

[RFC9110]

Fielding, R., Ed., Nottingham, M., Ed., and J. Reschke, Ed., "HTTP Semantics", STD 97, RFC 9110, DOI 10.17487/RFC9110, June 2022, <https://www.rfc-editor.org/rfc/rfc9110>.

[RFC9113]

Thomson, M., Ed. and C. Benfield, Ed., "HTTP/2", RFC 9113, DOI 10.17487/RFC9113, June 2022, <https://www.rfc-editor.org/rfc/rfc9113>.

[WHATWG-HTML]

WHATWG, "HTML Living Standard - Server-sent events", n.d., <https://html.spec.whatwg.org/multipage/server-sent-events.html>.

10.2. Informative References

[Connect]

Buf Technologies, "Connect Protocol Specification", n.d., <https://connectrpc.com/docs/protocol>.

[gRPC]

Google, "gRPC over HTTP/2", n.d., <https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md>.

[MCP]

Anthropic / AAIF, "Model Context Protocol Specification", n.d., <https://spec.modelcontextprotocol.io>.

[SignalR]

Microsoft, "SignalR Hub Protocol", n.d., <https://github.com/dotnet/aspnetcore/blob/main/src/SignalR/docs/specs/HubProtocol.md>.

[W3C-WEBTRANSPORT]

W3C, "WebTransport", n.d., <https://www.w3.org/TR/webtransport/>.