| Internet-Draft | IPv6 WPS Option | July 2026 |
| Yang, et al. | Expires 21 January 2027 | [Page] |
This document specifies the Wide-area Packet Spraying (WPS) option, an IPv6 option that can be carried in the Destination Options header or, where required, the Hop-by-Hop Options header. The option conveys the scheduling metadata that is needed to distribute the packets of a single high-volume flow across multiple parallel wide-area paths at the granularity of packet groups, and to restore the original packet ordering at the egress boundary of a controlled domain. The mechanism provides order-preserving multipath load balancing and bandwidth aggregation for wide-area interconnection of distributed computing sites, such as wide-area interconnects between Artificial Intelligence (AI) data centers, and is intended for use within limited domains.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 21 January 2027.¶
Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Per-flow load distribution over parallel wide-area links cannot exceed the capacity of one member link for any single flow, while per-packet spraying resolves the utilization problem but, over wide-area paths of unequal latency, introduces substantial packet reordering that severely degrades loss- and order-sensitive transports. This document therefore specifies Wide-area Packet Spraying (WPS), a mechanism that sprays traffic at an intermediate granularity: the packets of a flow are partitioned at the domain ingress into consecutive packet groups; all packets of one group are forwarded over the same path, while different groups are sprayed over different parallel paths. Intra-group ordering is preserved by path pinning, and inter-group ordering is restored at the domain egress using metadata carried in an IPv6 option defined in this document. The unit of spraying is thus the packet group rather than the individual packet, which retains the load-balancing benefit of spraying while bounding reordering.¶
WPS operates between an ingress node and an egress node at the boundaries of a single administrative domain. It does not modify end-to-end transport or congestion-control behavior, and it is transparent to the endpoints of the flows.¶
Training of large models increasingly spans several AI data centers, because a single site is constrained by power, cooling, floor space, and cluster size. Wide-area interconnection between such computing sites has therefore become a core part of the infrastructure for distributed training, and the loss of training efficiency attributable to the wide-area segment is commonly required to stay within a small budget (on the order of ten percent).¶
The traffic exchanged between such sites differs fundamentally from general Internet traffic and from intra-data-center traffic. It consists of a small number of concurrent flows, each of very high bandwidth and long duration, typically carried over RDMA-based transports, and it is highly sensitive to congestion, packet reordering, even small amounts of loss, and delay variation: loss triggers RDMA retransmission and rate reduction, which directly reduces the effective compute of a distributed training job.¶
Wide-area backbones commonly use per-flow hash-based Equal-Cost Multipath (ECMP) techniques that hash on invariant packet fields [RFC2991] [RFC2992] or on the IPv6 Flow Label [RFC6438]. For the traffic pattern above, this approach has the following deficiencies:¶
Several approaches can aggregate the bandwidth of multiple wide-area links for a single flow. This section records the trade-offs, so that the choice made in this document is documented rather than assumed.¶
Per-packet spraying maximizes link utilization but exposes the receiver to per-packet reordering. Over a wide area, the delay difference between parallel paths is far larger than within a data center, so the buffering and sorting effort at the receiver grows with the bandwidth-delay product, and the reordering directly disturbs transports that expect in-order delivery.¶
Flowlet-based switching relies on inter-packet gaps larger than the path delay difference to switch paths without reordering. Wide-area delay differences are on the order of milliseconds, and a large flow under sustained throughput exhibits almost no such gaps, so flowlet switching degenerates to per-flow hashing in this environment.¶
Entropy mechanisms such as the MPLS entropy label [RFC6790] and the IPv6 Flow Label [RFC6437] [RFC6438] improve the hash distribution among multiple flows but keep the flow as the placement unit; they do not address a single flow whose rate exceeds the capacity of one link. Distributing encapsulated or aggregated traffic over multiple paths by adding per-packet metadata is nevertheless established practice in these mechanisms; WPS extends the idea from per-flow placement to packet-group sequencing for wide-area multipath networks.¶
WPS distributes load at packet-group granularity, between the two extremes. Packets of a group follow one path and stay in order; reordering is confined to group boundaries and is repaired by the egress node with a bounded buffer. The group granularity is a tuning parameter that trades link balance against reordering effort (see Section 5).¶
The sequencing metadata is carried in an IPv6 option of an encapsulating header rather than in a new IP protocol header or a UDP encapsulation. Options are the native IPv6 extension mechanism [RFC8200], and the choice between the Destination Options header and the Hop-by-Hop Options header matches the two deployment modes described in Section 4.2. Encapsulation is required in any case, because an extension header cannot be inserted into a packet in flight (Section 4 of [RFC8200]); it also gives the egress node an unambiguous decapsulation point.¶
WPS is designed for limited domains [RFC8799]: administered networks such as the wide-area interconnect between AI data centers, in which the ingress and egress border nodes are under one operational authority and the parallel paths between them are known. WPS is not intended for use across the open Internet. The option-type encoding (Section 4.1) and the boundary filtering requirements (Section 11.1) together keep WPS packets confined to the domain.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The reference architecture is shown in Figure 1. An Ingress Node and an Egress Node are connected by multiple parallel wide-area member paths.¶
|<---------- WPS Domain ---------->|
| |
| ==== Path 1 ==== |
+--------+ +-+------+ ==== Path 2 ==== +--------+-+ +--------+
| Site |--| Ingress| ... | Egress |--| Site |
| A | | Node | ==== Path N ==== | Node | | B |
+--------+ +--------+ +----------+ +--------+
group-based group-order
spraying restoration
Operation proceeds in four steps:¶
Because packets of the same group traverse the same path, their relative order is preserved by normal forwarding. The reordering problem is thereby reduced from per-packet to per-group, which bounds the egress buffering requirement.¶
The WPS option has the following format:¶
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Option Type | Opt Data Len |S|E|R| Rsvd | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Flow ID | Ingress Node ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Egress Node ID | Group ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Ingress Port ID| Egress Port ID| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
The WPS option has an alignment requirement of 2n, which places all 16-bit fields of the Option Data on 16-bit boundaries. When the WPS option is the only option in an extension header, the resulting header is exactly 16 octets long, a multiple of 8 octets, and requires no padding options.¶
The assignment of Node ID and Port ID values within a WPS domain is an administrative matter for the domain operator and is outside the scope of this document.¶
Because an extension header cannot be added to a packet in flight (Section 4 of [RFC8200]), the Ingress Node MUST place the WPS option in an encapsulating IPv6 header [RFC2473]; it MUST NOT insert the option, or an options header carrying it, into a received packet.¶
The WPS option is designed to be examined only by the Egress Node. Therefore, the WPS option SHOULD be carried in a Destination Options header of the outer IPv6 header whose Destination Address identifies the Egress Node; transit nodes then forward the packet without examining the option.¶
Deployments in which transit nodes of the WPS domain need visibility of the scheduling metadata (for example, for path-aware queuing or telemetry) MAY carry the WPS option in a Hop-by-Hop Options header instead. Such deployments need to take into account the operational limitations of Hop-by-Hop option processing described in [RFC9673] and [RFC9098]: a source cannot assume that every node on the path processes the option, and such deployments MUST ensure that every node on the member paths either processes the option or forwards it unmodified without punting the packet to a slow path.¶
A packet MUST NOT carry the WPS option in both the Hop-by-Hop Options header and the Destination Options header, and MUST NOT carry more than one WPS option.¶
For each packet of an eligible flow, the Ingress Node performs the following steps:¶
The mapping of packet groups to member paths is a local scheduling decision. Uniform distribution (e.g., round-robin over equal-capacity paths) is RECOMMENDED as the default; weighted or load-aware scheduling based on measured path load MAY be used. Where the member paths are realized by ECMP inside the domain rather than by explicit tunnels, the Ingress Node MAY derive the outer IPv6 Flow Label from the Group ID so that on-path ECMP keeps all packets of a group on a single path [RFC6438].¶
Smaller group quanta yield finer load balancing but increase the reordering work at the egress; larger quanta approach per-flow behavior. The group quantum SHOULD be configurable, and the operator can set it according to the delay difference between the member paths and the buffering capacity of the Egress Node. An Ingress Node MUST bound the number of groups of one flow that are simultaneously in flight so that the egress reordering buffer requirement (Section 7) is bounded and Group ID wraparound ambiguity cannot occur.¶
Transit nodes of the WPS domain forward WPS packets according to normal IPv6 forwarding procedures. When the WPS option is carried in a Destination Options header, transit nodes do not examine it. When the option is carried in the Hop-by-Hop Options header and a transit node is configured to process that header [RFC9673] and recognizes the option, the node MAY use the option content for monitoring or for scheduling within the WPS domain, but it MUST NOT modify the option content.¶
A node that examines the option and does not recognize it discards the packet and sends an ICMPv6 Parameter Problem message, as required by the option-type encoding (Section 4.1). This contributes to keeping packets that carry the WPS option inside the WPS domain; it does not replace the boundary filtering of Section 11.1, because nodes that skip option processing forward such packets unchanged.¶
A transit node MUST NOT redistribute packets of the same outer flow across multiple paths on a per-packet basis; per-flow or flow-label-based placement, which keeps a group on a single path, is REQUIRED behavior for member-path internals.¶
Upon receiving a packet carrying the WPS option, the Egress Node performs the following steps:¶
Packets within one group arrive in order because they followed one path; no per-packet sorting is needed. If the release timer expires while groups are buffered, the Egress Node MUST advance NEG to the smallest buffered Group ID and release buffered groups in Group ID order. This bounds the latency penalty when a group boundary marker is lost or a member path stalls, so that a lost group does not stall the flow indefinitely. The timer value SHOULD be configurable and SHOULD default to a small multiple of the measured maximum latency skew between member paths.¶
If the reordering buffer for a flow is exhausted, the Egress Node MUST NOT discard buffered packets; it MUST instead release the buffered groups in Group ID order and advance NEG accordingly. Flow state SHOULD be reclaimed after a configurable idle period. If the R flag is 0, the Egress Node MAY skip steps 3-5 and forward packets on arrival.¶
Before forwarding a packet out of the WPS domain, the Egress Node MUST remove the outer encapsulation, including the extension header carrying the WPS option, restoring the packet as received at the Ingress Node.¶
WPS adds 56 octets to each packet: 40 octets of encapsulating IPv6 header and a 16-octet options header carrying the WPS option (Section 4.1). An implementation that uses WPS MUST account for these 56 octets when it performs Path MTU Discovery [RFC8201] or when it determines the effective MTU toward the Egress Node, following the tunnel MTU and fragmentation procedures of [RFC2473]. When PMTUD is not available, the implementation MUST use a conservative MTU estimate that accounts for the 56-octet overhead.¶
Operators of a WPS domain SHOULD provision the MTU of all member paths to accommodate this overhead for the largest expected ingress packet. When the traffic is RDMA, the service-side MTU or MSS configuration SHOULD be adjusted so that no fragmentation occurs on the wide-area segment.¶
WPS requires an upgrade only of the border devices of the two sites. When the Destination Options header is used, transit nodes need no knowledge of WPS at all; they forward ordinary IPv6 packets between the two border nodes.¶
WPS is most effective when the member paths have comparable capacity and bounded latency skew. Persistent capacity or latency asymmetry SHOULD be handled by weighting the group scheduler rather than by enlarging the egress reordering buffer.¶
Authorization and scheduling signaling of the WPS domain, if present, MAY be mapped to a high-priority forwarding class so that it is not blocked by ordinary traffic and remains timely and reliable.¶
Implementations SHOULD expose counters for groups transmitted and received per flow, out-of-order group arrivals, release-timer expiries, buffer occupancy high-water marks, and validation failures. These counters allow the operator to tune the group quantum, the release timer, and the scheduler weights.¶
The general security considerations for IPv6 extension headers discussed in [RFC9098] apply to the WPS option.¶
WPS is confined to a limited domain (Section 1.4). An operator MUST filter packets that carry the WPS option at the WPS domain boundary, in both directions: nodes at the boundary MUST be configured to prevent packets carrying the WPS option from leaking out of the domain and to discard packets arriving from outside the domain that carry the option. The option-type encoding, with the highest-order two bits set to "10", provides an additional line of defense: a node outside the domain that examines the option and does not recognize it discards the packet and returns an ICMPv6 Parameter Problem message. Because many nodes forward packets without examining options, this encoding alone does not guarantee confinement, which is why the boundary filtering requirement above is stated as a MUST.¶
Packets leaking to non-supporting nodes trigger ICMPv6 Parameter Problem messages. Nodes originating such messages are subject to the ICMPv6 rate-limiting requirements of [RFC4443], which limits the utility of the option as an ICMPv6 reflection vector.¶
The WPS option carries only scheduling metadata; it does not contain user payload, and it does not expose user data beyond what the encapsulated packet already exposes. The metadata (Flow ID and node and port identifiers) reveals limited information about the topology and the flow structure of the WPS domain to an on-path observer. An operator whose threat model includes on-path observation of the wide-area segment MAY protect that segment with encryption such as IPsec; such usage is outside the scope of this document.¶
An attacker able to inject packets with forged WPS options into the domain, or an on-path attacker that modifies the Group ID or Flow ID of packets, could cause an Egress Node to buffer packets needlessly, mis-order or advance flow state incorrectly, or exhaust reordering resources. Boundary filtering as required in Section 11.1 is the primary mitigation, and the Egress Node's validation of the Egress Node ID rejects packets that do not belong to it. Where the threat model includes on-path adversaries inside the domain, the encapsulation between Ingress and Egress Nodes can be protected with IPsec.¶
Egress Nodes MUST bound per-flow and aggregate reordering buffer usage, so that malicious or malfunctioning traffic consisting of incomplete packet groups cannot exhaust the buffer, and SHOULD police the rate of flow-state creation attributable to any single Ingress Node ID. The release-timer rule of Section 7 bounds the time for which a missing group can hold buffered packets.¶
IANA is requested to assign a new IPv6 option type from the "Destination Options and Hop-by-Hop Options" registry within the "Internet Protocol Version 6 (IPv6) Parameters" registry group, as follows:¶
| Hex Value | Binary Value (act/chg/rest) | Description | Reference |
|---|---|---|---|
| TBD1 (suggested 0x81) | 10 / 0 / 00001 | Wide-area Packet Spraying (WPS) | This document |
act = 10 requires a node that examines the option and does not recognize it to discard the packet and send an ICMPv6 Parameter Problem, Code 2, message to the packet's Source Address; chg = 0 indicates that the Option Data does not change en route [RFC8200].¶
The authors thank the members of the 6MAN working group for discussions on the safe use of IPv6 extension headers that informed the design choices in this document.¶