fann Y. Liu Internet-Draft X. Zhu Intended status: Standards Track ZTE Corporation Expires: 8 March 2027 4 September 2026 Fast Notification for Link Bandwidth draft-lz-fann-bandwidth-notification-00 Abstract This document proposes a data-plane-based method for rapidly advertising end-to-end path bandwidth information using a bitmap encoding. The mechanism enables fast load-balancing adjustments in AI/ML data center fabrics. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 8 March 2027. Copyright Notice Copyright (c) 2026 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Liu & Zhu Expires 8 March 2027 [Page 1] Internet-Draft Link Bandwidth Notification September 2026 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 2. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . 2 3. Link Bandwidth Notification Message . . . . . . . . . . . . . 3 4. Processing Procedures . . . . . . . . . . . . . . . . . . . . 4 5. Illustration Example . . . . . . . . . . . . . . . . . . . . 5 6. Security Considerations . . . . . . . . . . . . . . . . . . . 7 7. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 7 8. References . . . . . . . . . . . . . . . . . . . . . . . . . 7 8.1. Normative References . . . . . . . . . . . . . . . . . . 7 8.2. Informative References . . . . . . . . . . . . . . . . . 7 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 8 1. Introduction AI workloads are typically bandwidth-intensive and often generate multiple large data flows simultaneously. For optimized load balancing, it is necessary to obtain path capacity information across the network to dynamically distribute traffic across multiple equal- cost paths. [I-D.xu-idr-fare] introduces a new BGP Path Bandwidth Extended Community to carry end-to-end path bandwidth information within the data center fabric, enabling adaptive routing in CLOS networks. This approach relies on the control plane to process routing updates, resulting in relatively slow notification. [I-D.camarillo-rtgwg-lsn] defines a hardware-accelerated forwarding- plane notification protocol that propagates link reachability and congestion status across switches with sub-microsecond latency. This protocol currently does not carry any bandwidth information. Based on the bitmap concept introduced in [I-D.camarillo-rtgwg-lsn], this document proposes a data-plane-based method for rapidly advertising end-to-end path bandwidth information, which can be used to improve the effectiveness of load balancing. 2. Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here. Liu & Zhu Expires 8 March 2027 [Page 2] Internet-Draft Link Bandwidth Notification September 2026 3. Link Bandwidth Notification Message The message format is defined as follows: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | EtherType | OpCode | +-------------------------------+-------------------------------+ | Type |R|Msg| Rsv | Dev-Range | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ + | | | Bandwidth-Level-Bitmap | | (768 bits) | | | + +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Figure 1: Link Bandwidth Notification Message Format Field descriptions: EtherType: TBD1. OpCode: TBD2. Type: TBD3 - indicates that the message carries a Bandwidth-Level- Bitmap. R (1 bit): Reserved. MUST be set to 0 on transmission and ignored on receipt. Msg (2 bits): Message type (as defined in [I-D.camarillo-rtgwg-lsn]). Rsv (3 bits): Reserved. MUST be set to 0 on transmission and ignored on receipt. Dev-Range (6 bits): Determines the range of devices covered by the bitmap (same as in [I-D.camarillo-rtgwg-lsn]). Bandwidth-Level-Bitmap (768 bits): A bitmap of 768 bits, arranged contiguously in order of Leaf IDs. Each Leaf node is represented by a 3-bit bandwidth-level code. The encoding is defined as follows: Liu & Zhu Expires 8 March 2027 [Page 3] Internet-Draft Link Bandwidth Notification September 2026 +============+=================+==============================+ | 3-bit code | Bandwidth Level | Corresponding Link Bandwidth | +============+=================+==============================+ | 000 | Level 0 | Unreachable / link failure | +------------+-----------------+------------------------------+ | 001 | Level 1 | 10 Gbps | +------------+-----------------+------------------------------+ | 010 | Level 2 | 25 Gbps | +------------+-----------------+------------------------------+ | 011 | Level 3 | 50 Gbps | +------------+-----------------+------------------------------+ | 100 | Level 4 | 100 Gbps | +------------+-----------------+------------------------------+ | 101 | Level 5 | 200 Gbps | +------------+-----------------+------------------------------+ | 110 | Level 6 | 400 Gbps | +------------+-----------------+------------------------------+ | 111 | Level 7 | 800 Gbps and above | +------------+-----------------+------------------------------+ Table 1: Bandwidth Level Encoding 4. Processing Procedures The sending, receiving, and processing procedures are as follows: *Pre-configuration and information acquisition* Every leaf node within the fabric is assigned a unique Global Node ID. Additionally, a mapping between reachable IP prefixes and the leaf nodes that advertise them is required on each leaf node. This information can be obtained via a centralized controller or other methods, the details are out of the scope of this document. *Leaf sends bandwidth notification message* Each leaf sends a bandwidth notification message with its own 3-bit code set to the maximum value (i.e., 111). *Spine receives and updates the bitmap* Upon receiving bandwidth-level bitmaps from each leaf, the spine takes the minimum value between the received bitmap and its local port rate, and uses this minimum to update the bitmap. *Spine aggregates bitmaps and forwards* Liu & Zhu Expires 8 March 2027 [Page 4] Internet-Draft Link Bandwidth Notification September 2026 Based on the updated bitmaps for each leaf, the spine aggregates the bitmaps for each leaf (e.g., if there are two links towards the same leaf, the aggregated bandwidth is the sum of the bandwidths of those two links). The aggregated bitmap carries bandwidth information from this spine to each leaf. *Processing in the presence of a superspine* If a superspine exists in the network, it performs similar processing as the spine after receiving a bandwidth notification. The superspine compares the bandwidth from the received message with the bandwidth of its local ports to the spine, takes the minimum value to update the bitmap, aggregates bitmaps received from different spines for the same leaf, and then carries the aggregated bitmap in the bandwidth notification message to forward to other nodes. *Leaf receives and applies bandwidth information* After receiving the bandwidth notification message from the spine, the leaf node takes the minimum bandwidth value (i.e., compares the received bandwidth with the local link bandwidth to the spine) and then obtains the bandwidth capacity of each path to the IP prefix based on the Leaf-ID-to-IP-prefix mapping. The bandwidth notification messages are sent periodically at a configured interval. In addition, when a significant change occurs in the bandwidth or link state (e.g., link failure, link recovery, or bandwidth change), a notification message is triggered immediately. The resulting path bandwidth is used for optimized load balancing. For example, it can be used as a weight value when performing weighted ECMP load-balancing towards that prefix, or it can be combined with other link quality information (e.g., queue depth) to compute a link quality score, and the forwarding path is determined based on that score. Detailed load-balancing schemes are outside the scope of this document. 5. Illustration Example Liu & Zhu Expires 8 March 2027 [Page 5] Internet-Draft Link Bandwidth Notification September 2026 +---------+ +--------+ | Spine1 | | Spine2 | +-+-+-+-+-+ +-+-+-+-++ | | | | | | | | | | | +-----------------|-|-|-|-+ | | +------------+ | | | | | +--------+ +-+ | | | | | | | +----------------------------+ | | | | | | | +------------------+ | +----+ | | | | | +-------+ | | | | | | | | | | +-+---+-+ +--+--+-+ +-+--+--+ +--+--+-+ | Leaf1 | | Leaf2 | | Leaf3 | | Leaf4 | +-------+ +-------+ +-------+ +-------+ Figure 2: Example CLOS topology As shown in Figure 1, the CLOS topology contains two spine nodes (Spine1, Spine2) and four leaf nodes (Leaf1-Leaf4). Port 1, Port 2, Port 3, and Port 4 of Spine1 and Spine2 are connected to Leaf1, Leaf2, Leaf3, and Leaf4, respectively. All links are 100 Gbps, except that Spine1 connects to Leaf4 via a 2*50 Gbps LAG (logical total bandwidth 100 Gbps). The controller assigns a unique Leaf ID to each leaf and pre-installs the mapping between Leaf IDs and IP prefixes. *Step 1: Leaf sends initial bandwidth-level notification messages.* Leaf1-Leaf4 each send bandwidth-level notification messages to Spine1 and Spine2. For example, Leaf1 sends a bitmap where its own 3 bits are 111 (maximum), and the remaining bits for the other three leaves are 000 (unreachable initially). Thus Leaf1's bitmap is 111000000000. *Step 2: Spine receives and updates bitmaps* Spine1 and Spine2 receive notifications from all leaves. They take the minimum between the received bitmap and their local port bandwidth. For Spine1's ports 1-3 and all ports of Spine2, the available bandwidth is updated to 100 Gbps (code 100). For Spine1 port 3 (the LAG to Leaf4), the available bandwidth is updated to 50 Gbps (code 011). The updated bitmaps are stored locally. For example, Spine1 port 0 stores a bitmap 100000000000. *Step 3: Spine aggregates and sends bitmaps* Liu & Zhu Expires 8 March 2027 [Page 6] Internet-Draft Link Bandwidth Notification September 2026 Spine1 and Spine2 aggregate the locally maintained bitmaps for Leaf1 through Leaf4 based on routing reachability. Spine1 aggregates to 100100100011, meaning that from Spine1, the available bandwidth to Leaf1-Leaf3 is 100 Gbps (code 100), and to Leaf4 is 50 Gbps (code 011). Similarly, Spine2 aggregates to 100100100100, meaning that to all leaf nodes the available bandwidth is 100 Gbps (code 100). *Step 4: Leaf updates its port bitmaps* Leaf1 receives from Spine1 the bitmap 100100100011 and from Spine2 the bitmap 100100100100. It takes the minimum bandwidth value for each path (which remains unchanged in this case), and updates its local forwarding table accordingly. The resulting path bandwidth information is then used for weighted load balancing. 6. Security Considerations The bandwidth notification mechanism defined in this document is intended solely for use within controlled, single-domain AI/ML data center fabrics. To mitigate potential spoofing and denial-of-service attacks, switches MUST implement strict port-level filtering. Bandwidth notification messages MUST be dropped unconditionally on any port facing a server, host, or external network. Processing of these messages MUST only be enabled on trusted, switch-to-switch infrastructure links. 7. IANA Considerations TBA 8. References 8.1. Normative References [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . [RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, . 8.2. Informative References [I-D.camarillo-rtgwg-lsn] Camarillo, P., Filsfils, C., Chachmon, N., Iny, O., Su, Y., and R. Jiang, "Lightspeed Notification Protocol", Work Liu & Zhu Expires 8 March 2027 [Page 7] Internet-Draft Link Bandwidth Notification September 2026 in Progress, Internet-Draft, draft-camarillo-rtgwg-lsn-00, 2 March 2026, . [I-D.xu-idr-fare] Xu, X., Hegde, S., Patel, K., He, Z., and H. Wu, "Fully Adaptive Routing Ethernet using BGP", Work in Progress, Internet-Draft, draft-xu-idr-fare-08, 17 August 2026, . Authors' Addresses Yao Liu ZTE Corporation Email: liu.yao71@zte.com.cn Xiangyang Zhu ZTE Corporation Email: zhu.xiangyang@zte.com.cn Liu & Zhu Expires 8 March 2027 [Page 8]