Fast Notification for Link Bandwidth

Introduction AI workloads are typically bandwidth-intensive and often generate multiple large data flows simultaneously. For optimized load balancing, it is necessary to obtain path capacity information across the network to dynamically distribute traffic across multiple equal-cost paths. introduces a new BGP Path Bandwidth Extended Community to carry end-to-end path bandwidth information within the data center fabric, enabling adaptive routing in CLOS networks. This approach relies on the control plane to process routing updates, resulting in relatively slow notification. defines a hardware-accelerated forwarding-plane notification protocol that propagates link reachability and congestion status across switches with sub-microsecond latency. This protocol currently does not carry any bandwidth information. Based on the bitmap concept introduced in , this document proposes a data-plane-based method for rapidly advertising end-to-end path bandwidth information, which can be used to improve the effectiveness of load balancing.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Link Bandwidth Notification Message The message format is defined as follows:

Link Bandwidth Notification Message Format Field descriptions:

EtherType:: TBD1.
OpCode:: TBD2.
Type:: TBD3 - indicates that the message carries a Bandwidth-Level-Bitmap.
R (1 bit):: Reserved. MUST be set to 0 on transmission and ignored on receipt.
Msg (2 bits):: Message type (as defined in ).
Rsv (3 bits):: Reserved. MUST be set to 0 on transmission and ignored on receipt.
Dev-Range (6 bits):: Determines the range of devices covered by the bitmap (same as in ).
Bandwidth-Level-Bitmap (768 bits):: A bitmap of 768 bits, arranged contiguously in order of Leaf IDs. Each Leaf node is represented by a 3-bit bandwidth-level code. The encoding is defined as follows:

Bandwidth Level Encoding

3-bit code	Bandwidth Level	Corresponding Link Bandwidth
000	Level 0	Unreachable / link failure
001	Level 1	10 Gbps
010	Level 2	25 Gbps
011	Level 3	50 Gbps
100	Level 4	100 Gbps
101	Level 5	200 Gbps
110	Level 6	400 Gbps
111	Level 7	800 Gbps and above

Processing Procedures The sending, receiving, and processing procedures are as follows: Pre-configuration and information acquisition Every leaf node within the fabric is assigned a unique Global Node ID. Additionally, a mapping between reachable IP prefixes and the leaf nodes that advertise them is required on each leaf node. This information can be obtained via a centralized controller or other methods, the details are out of the scope of this document. Leaf sends bandwidth notification message Each leaf sends a bandwidth notification message with its own 3-bit code set to the maximum value (i.e., 111). Spine receives and updates the bitmap Upon receiving bandwidth-level bitmaps from each leaf, the spine takes the minimum value between the received bitmap and its local port rate, and uses this minimum to update the bitmap. Spine aggregates bitmaps and forwards Based on the updated bitmaps for each leaf, the spine aggregates the bitmaps for each leaf (e.g., if there are two links towards the same leaf, the aggregated bandwidth is the sum of the bandwidths of those two links). The aggregated bitmap carries bandwidth information from this spine to each leaf. Processing in the presence of a superspine If a superspine exists in the network, it performs similar processing as the spine after receiving a bandwidth notification. The superspine compares the bandwidth from the received message with the bandwidth of its local ports to the spine, takes the minimum value to update the bitmap, aggregates bitmaps received from different spines for the same leaf, and then carries the aggregated bitmap in the bandwidth notification message to forward to other nodes. Leaf receives and applies bandwidth information After receiving the bandwidth notification message from the spine, the leaf node takes the minimum bandwidth value (i.e., compares the received bandwidth with the local link bandwidth to the spine) and then obtains the bandwidth capacity of each path to the IP prefix based on the Leaf-ID-to-IP-prefix mapping. The bandwidth notification messages are sent periodically at a configured interval. In addition, when a significant change occurs in the bandwidth or link state (e.g., link failure, link recovery, or bandwidth change), a notification message is triggered immediately. The resulting path bandwidth is used for optimized load balancing. For example, it can be used as a weight value when performing weighted ECMP load-balancing towards that prefix, or it can be combined with other link quality information (e.g., queue depth) to compute a link quality score, and the forwarding path is determined based on that score. Detailed load-balancing schemes are outside the scope of this document.

Illustration Example

Example CLOS topology As shown in Figure 1, the CLOS topology contains two spine nodes (Spine1, Spine2) and four leaf nodes (Leaf1-Leaf4). Port 1, Port 2, Port 3, and Port 4 of Spine1 and Spine2 are connected to Leaf1, Leaf2, Leaf3, and Leaf4, respectively. All links are 100 Gbps, except that Spine1 connects to Leaf4 via a 2*50 Gbps LAG (logical total bandwidth 100 Gbps). The controller assigns a unique Leaf ID to each leaf and pre-installs the mapping between Leaf IDs and IP prefixes. Step 1: Leaf sends initial bandwidth-level notification messages. Leaf1-Leaf4 each send bandwidth-level notification messages to Spine1 and Spine2. For example, Leaf1 sends a bitmap where its own 3 bits are 111 (maximum), and the remaining bits for the other three leaves are 000 (unreachable initially). Thus Leaf1's bitmap is 111000000000. Step 2: Spine receives and updates bitmaps Spine1 and Spine2 receive notifications from all leaves. They take the minimum between the received bitmap and their local port bandwidth. For Spine1's ports 1-3 and all ports of Spine2, the available bandwidth is updated to 100 Gbps (code 100). For Spine1 port 3 (the LAG to Leaf4), the available bandwidth is updated to 50 Gbps (code 011). The updated bitmaps are stored locally. For example, Spine1 port 0 stores a bitmap 100000000000. Step 3: Spine aggregates and sends bitmaps Spine1 and Spine2 aggregate the locally maintained bitmaps for Leaf1 through Leaf4 based on routing reachability. Spine1 aggregates to 100100100011, meaning that from Spine1, the available bandwidth to Leaf1-Leaf3 is 100 Gbps (code 100), and to Leaf4 is 50 Gbps (code 011). Similarly, Spine2 aggregates to 100100100100, meaning that to all leaf nodes the available bandwidth is 100 Gbps (code 100). Step 4: Leaf updates its port bitmaps Leaf1 receives from Spine1 the bitmap 100100100011 and from Spine2 the bitmap 100100100100. It takes the minimum bandwidth value for each path (which remains unchanged in this case), and updates its local forwarding table accordingly. The resulting path bandwidth information is then used for weighted load balancing.

Security Considerations The bandwidth notification mechanism defined in this document is intended solely for use within controlled, single-domain AI/ML data center fabrics. To mitigate potential spoofing and denial-of-service attacks, switches MUST implement strict port-level filtering. Bandwidth notification messages MUST be dropped unconditionally on any port facing a server, host, or external network. Processing of these messages MUST only be enabled on trusted, switch-to-switch infrastructure links.

IANA Considerations TBA