]> China Telecom

109, West Zhongshan Road, Tianhe District Guangzhou Guangzhou 510000 CN hujy5@chinatelecom.cn

Routing Routing Area Working Group RFC This document defines the precise use of ECN in WAN. With the growing demand for AI computing power, the computational capacity of a single Artificial Intelligence Data Center (AIDC) can no longer meet the requirements of large-scale model training. This has led to the emergence of cross-AIDC distributed model training, driving the need for transmitting RoCEv2 packets over WAN networks. AI training is highly sensitive to network packet loss, where even minimal packet loss can significantly degrade training efficiency. Additionally, elephant flows and extreme concurrent traffic impose higher demands on network performance. ECN achieves active feedback of network congestion by setting ECN flag bits in the header of IP packets, which is an effective traffic control method. RFC 6040 introduces the application of ECN in WAN. However, due to the much higher end-to-end delay in WAN than in DC, and the frequent occurrence of instantaneous traffic bursts in WAN, it is easy to trigger ECN at the wrong time. This document focuses on the precise use of ECN in WAN, by introducing different reactions of ECN in different WAN transmission scenarios. It updates RFC 6040 to specify a domain-local early-warning semantics for the ECT(1) codepoint within managed WAN infrastructure.

Introduction The rapid growth of AI computing power, particularly for large-scale model training, has necessitated distributed training across multiple Artificial Intelligence Data Centers (AIDCs). This shift has increased the demand for reliable and high-performance transmission of RoCEv2 (RDMA over Converged Ethernet version 2) traffic over the WAN. However, AI workloads are highly sensitive to network congestion and packet loss, even minor packet drops can significantly degrade training efficiency. Due to the long links and significant end-to-end latency in wide area networks, traditional congestion control mechanisms may not be effective in a timely manner. They are insufficient for AI workloads due to their reactive nature and inability to ensure near-zero packet loss. To address these challenges, this draft explores the precise utilization of Explicit Congestion Notification (ECN) in WAN environments, particularly for RoCEv2 over IP tunnels. ECN enables proactive congestion signaling by marking packets instead of dropping them, allowing endpoints to adjust transmission rates before congestion escalates. However, traditional ECN implementations face challenges in WAN scenarios, including inconsistent ECN propagation across tunnel boundaries and inefficient congestion response mechanisms. This work focuses on optimizing ECN for lossless RoCEv2 transmission in WANs by: 1. Ensuring Accurate ECN Propagation: Defining rules for consistent ECN field handling across IP-in-IP tunnels to prevent packet loss. 2. Enhancing Congestion Feedback: Adjust the sending rate within a small range of the wide area network to reduce the impact of latency on end-to-end communication. 3. Supporting Multi-Level Congestion Signaling: Extending ECN to differentiate between varying congestion severities, improving responsiveness for AI traffic. By refining ECN mechanisms for WAN environments, this approach enhances network efficiency for distributed AI training while maintaining backward compatibility with existing protocols. The proposed framework provides a scalable and reliable solution for future large-scale distributed computing applications.

Relationship to Other Work This document is scoped to ECN codepoint semantics and their propagation across IP-in-IP tunnels in a WAN. The early-warning signal (ECT(1)) defined here is distinct from the Pre-Congestion Notification (PCN) concept of , which is encoded on Diffserv codepoints rather than the ECN field; this document therefore does not use the PCN term. Re-use of ECT(1) within a single managed WAN domain follows the flexibility permitted by and the guidance for specifying alternate ECN semantics in (BCP 124). This work complements the "Fast CNP with Proxy" concept of , which is being progressed in the FANN working group and addresses how a congested node can notify a traffic sender when it cannot understand or deliver the notification directly. This document instead keeps early-warning congestion local to the ingress domain, and propagates only the severe congestion codepoint (CE) toward the end host. The two approaches address different problems and may be combined. This document is also related to, but independent from, the Fast Network Notifications (FANN) framework. FANN defines a general framework for fast, low-latency network notifications (congestion level, queue length, latency). This document can be regarded as an ECN-oriented use case that provides fast congestion feedback through the ECN codepoints and an in-domain CNP. The Fast CNP without a proxy, defined in , is sent by a congested node directly to the traffic sender in RoCEv2 networks. This document is complementary to it: the in-domain CNP described here targets the ingress PE of the WAN domain instead of the traffic sender, and keeps early-warning handling internal to the domain. The ECT(1) codepoint is also used by the Low Latency, Low Loss, and Scalable Throughput (L4S) service as the identifier for L4S-capable traffic. The early-warning semantics defined in this document therefore conflict with L4S when both are active in the same domain. This mechanism is intended to be deployed only within a single managed WAN domain in which L4S traffic and L4S marking are not enabled; at the domain boundary an early-warning mark MUST NOT be propagated to nodes that do not understand it (see ). Operators that enable L4S in a domain MUST NOT enable this mechanism on the same path. The tunnel behavior defined here is consistent with the guidelines for adding congestion notification to protocols that encapsulate IP in : the outer IP header carries the ECN codepoints used for congestion signaling, while the inner RoCEv2 payload is left unmodified. More accurate ECN feedback for TCP, e.g., AccECN , improves the granularity of end-to-end ECN feedback at the transport layer. The mechanism in this document is orthogonal to it: it operates at the network layer within a managed WAN domain and does not change the transport-layer ECN feedback loop.

Requirements Language The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document is an Informational contribution; the keywords precisely describe the behaviour of the mechanism and do not impose normative requirements beyond the scope of this document.

AIDC: Artificial Intelligence Data Center RoCEv2: RDMA over Converged Ethernet version 2 ECN: Explicit Congestion Notification CNP: Congestion Notification Packet

ECN Mechanism for WANs In WANs, tunneling is a fundamental technique used to encapsulate and transport data packets across different network domains while maintaining security, performance, and compatibility. Tunneling works by embedding an original packet (the inner payload) within a new packet (the outer header), allowing it to traverse intermediate networks that may not natively support the original protocol. ECN, as a traditional congestion notification mechanism, has also been extended from DC to WAN. introduces how to label and use ECN mechanisms in tunnels, which are divided into tunnel ingress behavior and tunnel egress behavior. Each behavior contains two encapsulation modes: a "compatibility mode," which is for backward compatibility with tunnel decapsulators that do not comprehend ECN, and a REQUIRED "normal mode." The detail of ingress behavior is shown below:

Early-Warning ECN: Encapsulation Behaviours

For the decapsulation behavior, detail is shown below:

Early-Warning ECN: Decapsulation Behaviours (!!!)| | ECT(0) | ECT(0) | ECT(0) | light CE* | CE | | ECT(1) | ECT(1) | ECT(1) (!) | light CE* | CE | | CE | CE | CE | CE(!!!)| CE | +---------+---------+------------+------------+------------+ * "light CE" denotes the early-warning congestion state defined in this document within a managed WAN domain. It is a local state at the decapsulating PE: the outer ECT(1) mark is consumed to trigger in-domain CNP handling, and the inner ECN field is left unchanged, so the local semantics are not exposed to nodes outside the domain. ]]>

In the decapsulation table above, '(!)' and '(!!!)' follow the notation of for combinations of inner and outer ECN fields that were currently unused (CU) when it was specified, for which it also specifies that a decapsulator SHOULD log the event. The 'light CE' entries deviate from only in the column for an arriving outer ECT(1) header: within a managed WAN domain, an arriving outer ECT(1) mark denotes the early-warning state defined in this document. The decapsulating PE consumes this mark to trigger in-domain CNP handling and leaves the inner ECN field unchanged, so that the domain-local ECT(1) semantics are not propagated beyond the domain boundary. ECT(0) and ECT(1) can both indicate the same degree of congestion marking (such as "not congestion marked") according to the reasoning above. However, it also makes it possible to construct future schemes in which ECT(1) can represent other situations in WAN scenarios.

Two-Threshold ECN Mechanism for WAN The high latency and bursty nature of WAN links introduce significant challenges for timely and accurate congestion management. Traditional single-threshold ECN or packet-drop mechanisms often result in delayed feedback, causing over-correction (global synchronization) or under-correction (persistent congestion). To address this, this draft proposes a Two-Threshold ECN Mechanism specifically designed for WAN environments carrying latency-sensitive, loss-averse traffic like RoCEv2 for AI training.

Mechanism Overview This mechanism redefines the use of the ECT(1) codepoint within a controlled domain (e.g., a provider's WAN core), leveraging the flexibility permitted by . Network devices (e.g., routers, switches) supporting this mechanism are configured with two queue occupancy thresholds: a Lower Threshold (T1) and a Higher Threshold (T2). ECT(1) as "Pre-Congestion" or "Early Warning" Signal: When the average queue length exceeds T1, the device interprets this as incipient or light congestion. Packets with an outer IP header ECN field of ECT(0) or ECT(1) are remarked to ECT(1) with a probability that increases linearly with the queue length. This ECT(1) marking is defined within this WAN domain as an early-warning congestion signal (referred to below simply as the "early-warning signal"). Its purpose is to signal an impending congestion condition before queues build to a level that would impact latency or cause loss. CE as "Severe Congestion" Signal: When the average queue length exceeds T2, the device interprets this as severe congestion requiring immediate action. Packets are marked with the CE codepoint following a standard RED-like algorithm. This signal mandates a direct and measurable reduction in the data sender's transmission rate.

Congestion Notification Packet (CNP) and Ingress PE Action A critical component of this mechanism is the generation and processing of a Congestion Notification Packet (CNP). This is a control packet generated by the congested device (or a network controller monitoring it) and sent to the tunnel ingress Provider Edge (PE) device -- the point where the traffic entered the WAN domain. CNP Format: this specification does not mandate a new CNP wire format; the CNP is carried as a control message between the congested device and the ingress PE over an existing channel (for example, a UDP-based or in-band control channel). At minimum, the CNP MUST identify the affected flow (e.g., a 5-tuple or a flow identifier), the congestion level (early or severe), and include a value or timestamp sufficient for rate limiting. The ingress PE MUST rate-limit CNP handling to avoid reacting to transient micro-bursts, and SHOULD discard CNPs inconsistent with the locally observed queue state.

Upon reaching T1 (ECT(1) marking): The congested device generates and sends a CNP to the ingress PE. This CNP identifies the affected flow (e.g., via 5-tuple) and indicates a light congestion event. Ingress PE Action: Upon receiving this CNP, the ingress PE MAY take proactive measures to alleviate the impending congestion without involving the end host. This can include: Local Rate Adjustment: Slightly reducing the transmission rate for the identified flow into the tunnel. Traffic Rerouting: Dynamically steering the flow to an alternative, less congested path within the WAN if available (e.g., using SRv6 policy). ECN Propagation at the Egress: independently of the CNP, when a packet carrying the outer ECT(1) early-warning mark reaches the decapsulating (egress) PE, that PE does NOT copy the mark to the inner IP header (following a modified "pipe model" logic for this codepoint). The end host remains unaware of this early warning, preventing an over-reaction from a distant sender whose feedback loop is delayed by the WAN RTT.

Upon reaching T2 (CE marking): The congested device generates and sends a CNP to the ingress PE, now indicating a severe congestion event. Ingress PE Action: Upon receiving this CNP, the ingress PE MUST intensify its local congestion control for the affected flow, for example by applying stronger versions of the Local Rate Adjustment and Traffic Rerouting actions described for the T1 case. Independently of this, the decapsulating (egress) PE performs standard RFC 6040 "normal mode" decapsulation: the CE codepoint from the outer header is copied to the inner IP header, so that the end-to-end transport's congestion control is engaged. The packet, now with CE set in the inner header, is forwarded to the receiver. The receiver's transport (e.g., RoCEv2) then feeds this congestion signal back to the original sender, which MUST reduce its transmission rate according to its congestion control algorithm.

Deployment and Compatibility Considerations Backward Compatibility: Devices not implementing this two-threshold mechanism will treat ECT(1) as equivalent to ECT(0) per , and will process CE normally. This ensures safe co-existence and incremental deployment. Domain of Application: This mechanism is intended for use within a managed WAN domain (e.g., a single provider's core). The re-definition of the ECT(1) semantics is a local policy, confined to the outer tunnel header inside the domain. At the egress PE leaving this domain, standard RFC 6040 rules apply for forwarding packets into external networks. Egress Behavior at the Domain Boundary: when a packet traverses the egress PE that leaves the managed WAN domain, the outer header is stripped and the inner ECN field is preserved, following the standard RFC 6040 normal-mode decapsulation rules (see the decapsulation table above). In particular, an early-warning (ECT(1)) mark applied to the outer header inside the domain MUST NOT be propagated to an inner endpoint that does not understand it: the egress PE consumes that mark for in-domain CNP handling and leaves the inner ECN field unchanged, so that the local ECT(1) semantics are not leaked to nodes outside the domain. CE is asserted only when severe congestion applies. Threshold Tuning: The values of T1 and T2 are critical and should be set based on link capacity, typical traffic profiles, and the desired latency-loss trade-off for the target applications (e.g., AI training). T1 should be set low enough to provide meaningful early warning but high enough to avoid triggering on transient micro-bursts. T2 MUST be set strictly greater than T1, so that the early-warning and severe congestion regions are clearly separated.

IANA Considerations No IANA actions are required by this specification; it does not define new protocol elements, codepoints, or registries.

Security Considerations The proposed enhancements introduce new mechanisms that must be evaluated for potential security vulnerabilities. This section expands upon the security considerations of and within the context of this draft.

Threats Related to the Two-Threshold Mechanism CNP Spoofing and Forgery: An attacker could generate malicious CNPs and send them to an ingress PE, falsely indicating congestion. This could trigger unnecessary rate reduction or rerouting, leading to denial-of-service (performance degradation) for legitimate flows or manipulation of traffic paths for interception. Threshold Manipulation: An on-path attacker with access to network device configuration could alter the T1 or T2 thresholds. Lowering T1 excessively would cause frequent ECT(1) marking and CNP generation, leading to under-utilization of the link. Raising T2 excessively could suppress legitimate CE signals, leading to bufferbloat and packet loss. ECN Field Tampering within the Tunnel: As noted in and , the outer ECN field is mutable. An attacker within the WAN could erase CE marks to hide congestion from the sender, or could set false CE/ECT(1) marks to artificially throttle flows. The two-threshold mechanism's use of ECT(1) as a significant signal creates a new vector for manipulation.

Covert Channel Considerations explicitly relaxed earlier restrictions on the covert channel bandwidth across tunnels, deeming a 2-bit per packet channel manageable. This mechanism does not alter that fundamental assessment. However, the specific semantics where the ingress PE does not propagate ECT(1) outwards but does act on a CNP could theoretically be exploited by a colluding ingress and egress point to encode information. This is considered a manageable risk within a single administrative domain.

References Normative References Informative References ZTE Corporation China Mobile China Telecom ZTE Corporation ZTE Corporation China Mobile China Mobile China Mobile China Telecom

Contributors Thanks to all the contributors.