What Is Scale-Out Networking in AI Data Centers?

1. What Is Scale-Out Networking?

Scale-out networking is an architecture that connects multiple independent servers, GPU systems, and racks through a high-performance network fabric. Instead of increasing the resources inside one server, scale-out expands computing capacity by adding more connected nodes.

In AI data centers, each server may contain multiple GPUs, local memory, CPUs, and high-speed network interface cards (NICs). The scale-out network links these servers so that GPUs across different machines can exchange model parameters, gradients, activations, and other training data.

This architecture is essential for distributed AI training, large-scale inference, scientific computing, and other workloads that require more accelerator resources than a single computing system can provide.

2. Why Do AI Data Centers Need Scale-Out Networks?

Modern AI models require substantial computing capacity and memory. A single GPU server may contain several accelerators, but large training jobs often require dozens, hundreds, or thousands of GPUs distributed across multiple servers.

Adding more GPU servers increases the available computing resources, but it also creates additional communication requirements. The network must move data between nodes without becoming a bottleneck.

Without sufficient network capacity, adding more GPUs does not necessarily produce proportional improvements in training performance. Communication overhead, congestion, and synchronization delays can limit the benefits of additional computing resources.

3. How Does a Scale-Out Network Work?

A scale-out network connects GPU servers through NICs and network switches. Each server sends and receives data through its network interfaces, while the switching fabric provides paths between different nodes.

A simplified communication path looks like this:

When a GPU on Server A needs to exchange data with a GPU on Server B, the communication software initiates a transfer through the supported networking stack. The NIC handles network transmission, switches forward packets through the fabric, and the destination NIC delivers the data to the receiving system.

With compatible RDMA technology, data can be transferred between GPU memory regions with reduced CPU involvement. GPUDirect RDMA can further enable supported NICs to access GPU memory directly, avoiding unnecessary intermediate copies through host memory.

4. Leaf-Spine Architecture in AI Data Centers

Leaf-Spine is a common network topology for large-scale data centers. It uses two main switch layers to provide scalable connectivity between servers and network infrastructure.

Leaf Switches

Leaf switches connect directly to GPU servers, storage systems, and other network endpoints. They form the access layer of the fabric and provide uplinks to multiple spine switches.

In an AI cluster, each GPU server may connect to one or more leaf switches. Multiple connections can provide additional bandwidth or support network resilience, depending on the design.

Spine Switches

Spine switches form the central switching layer. Each leaf switch connects to the spine layer, creating multiple possible paths between servers attached to different leaf switches.

This structure supports horizontal expansion. When more servers are added, additional leaf switches can be introduced, while spine capacity can be expanded to accommodate the increased traffic demand.

Leaf-Spine is not the only possible topology. Large AI fabrics may also use rail-optimized, multi-plane, or other Clos-based designs. The appropriate structure depends on accelerator layout, NIC placement, traffic patterns, switch capacity, and the collective communication software.

5. InfiniBand and RoCEv2: Two Major Scale-Out Technologies

InfiniBand and RoCEv2 are widely used technologies for high-performance communication between GPU servers. Both support RDMA-based data transfers, but they use different network approaches.

InfiniBand

InfiniBand is a high-performance interconnect technology designed for low-latency communication and high-throughput computing environments. It is widely used in high-performance computing and large AI training clusters.

Its architecture includes specialized host channel adapters, switches, and fabric management components. RDMA is an integral part of the InfiniBand ecosystem, making it suitable for communication-intensive distributed workloads.

RoCEv2

RoCEv2, or RDMA over Converged Ethernet version 2, enables RDMA communication over Ethernet networks. It encapsulates RDMA traffic within UDP/IP packets, allowing compatible network adapters to exchange data through an Ethernet switching fabric.

RoCEv2 can integrate with Ethernet-based data center infrastructure, but high-performance deployments require careful engineering. Congestion control, traffic management, queue behavior, and network configuration can all influence performance.

Neither technology is automatically the right choice for every AI deployment. Selection depends on performance requirements, existing infrastructure, operational expertise, software support, and the scale of the cluster.

6. The Role of NICs, RDMA, and GPUDirect RDMA

Network interface cards are essential components in a scale-out AI cluster. They connect GPU servers to the network fabric and handle the transmission and reception of data between nodes.

High-Speed Network Interface Cards

AI servers may use high-bandwidth NICs or SuperNICs to support intensive GPU-to-GPU communication. The number of network adapters and their port speeds determine how much traffic each server can send and receive.

Some platforms assign separate network interfaces to different GPU groups or communication rails. This can help distribute traffic across multiple network paths, depending on the system architecture.

Remote Direct Memory Access

RDMA allows one system to access memory in another system without requiring the remote CPU to handle every data transfer operation. This reduces CPU overhead and can improve efficiency for large-scale communication workloads.

GPUDirect RDMA

GPUDirect RDMA extends this approach to compatible GPU memory. A supported network adapter can transfer data directly between the network and GPU memory, reducing the need for intermediate copies through host memory.

Actual performance depends on the GPU platform, NIC, PCIe topology, drivers, firmware, and communication software. Direct memory access should not be assumed to work identically across all server configurations.

7. How Scale-Out Networking Supports Distributed AI Training

Distributed training divides a workload across multiple GPUs. When these GPUs are located in different servers, the scale-out network carries the data required to coordinate their work.

Several communication operations are especially important:

Communication libraries such as NVIDIA NCCL coordinate these operations across GPUs. The library can use available intra-server and inter-server communication paths to move data between participants.

For example, in data-parallel training, GPUs calculate gradients locally and then exchange information through an All-Reduce operation. When GPUs are distributed across servers, the scale-out network carries the inter-node portion of that communication.

In mixture-of-experts models, All-to-All communication can become important because tokens may need to move between GPUs hosting different experts. This creates a different traffic pattern from conventional data-parallel training.

8. Network Bandwidth, Oversubscription, and Congestion

A scale-out network must provide enough capacity for the traffic generated by all connected GPU servers. The total number of GPU ports is only one part of the design. Uplink bandwidth, switch capacity, traffic distribution, and communication patterns also matter.

Network Bandwidth

Network bandwidth determines how much data can be transferred over a link within a given period. AI clusters use high-speed connections because distributed training and inference can generate substantial east-west traffic between servers.

Oversubscription Ratio

Oversubscription describes the relationship between the total bandwidth available from connected endpoints and the bandwidth available through their shared uplinks.

For example, if the aggregate server-facing bandwidth is 800 Gbps and the corresponding uplink capacity is 400 Gbps, the nominal oversubscription ratio is 2:1.

A lower oversubscription ratio can provide more available uplink capacity, but the appropriate design depends on expected traffic patterns, workload behavior, cost, and the level of performance required.

Network Congestion

Congestion occurs when multiple traffic flows compete for limited network resources. In AI clusters, many GPUs may transmit data at similar times, creating bursts of traffic across shared links.

Congestion can increase latency, reduce effective throughput, and cause communication operations to take longer. Network design therefore needs to consider traffic distribution, congestion management, queueing behavior, and telemetry.

9. Why Optical Connectivity Matters in Scale-Out Networks

As GPU clusters expand across racks, the network needs to connect a growing number of servers and switches. High-speed optical connectivity helps support these links where electrical cable reach, routing density, or signal integrity becomes a limiting factor.

Optical transceivers convert electrical signals from network equipment into optical signals for transmission through fiber. At the receiving end, the optical signal is converted back into an electrical signal for the destination device.

Fiber links are particularly important for switch-to-switch connections and other data center links that require higher bandwidth over longer distances than practical copper connections can support.

400G Connectivity

400G interfaces are used in high-speed data center networks and can support switch uplinks, server connections, and other network links. The specific optical module or cable depends on the required reach, fiber type, connector, and equipment interface.

800G Connectivity

800G increases the aggregate capacity available through a network port. It is relevant to newer AI fabrics where large numbers of GPU servers generate substantial east-west traffic.

1.6T Connectivity

1.6T optical connectivity represents another step in increasing network port capacity. It can support future high-density switching platforms and larger AI network fabrics as compatible switches, NICs, and optical modules become available.

The move from 400G to 800G or 1.6T is not simply a matter of replacing optical modules. Switch capacity, NIC support, port configuration, cabling, fiber infrastructure, and network topology must all be considered.

10. How to Design a Scalable AI Scale-Out Fabric

A scalable AI network should be designed around the communication requirements of the workload, rather than relying only on the number of GPU servers.

Determine the Compute Scale

Identify the number of servers, GPUs per server, NIC bandwidth, and expected cluster growth. These factors establish the initial network capacity requirements.

Understand the Traffic Pattern

Data-parallel, tensor-parallel, pipeline-parallel, and expert-parallel workloads generate different communication patterns. The network should be sized for the actual traffic behavior of the intended applications.

Select the Network Fabric

Evaluate InfiniBand or RoCEv2 according to performance targets, infrastructure preferences, operational requirements, and software compatibility.

Plan Leaf-Spine Capacity

Determine the number of leaf and spine switches, uplink capacity, port utilization, and oversubscription targets. Ensure that the fabric can support expected traffic between compute nodes.

Choose Suitable Optical and Electrical Links

Use DAC or AEC for supported short-reach connections and AOC or optical transceivers where fiber connectivity is more appropriate. Match the product to the required data rate, reach, connector, and equipment platform.

Monitor and Validate Performance

Test real communication workloads and monitor throughput, latency, congestion, packet behavior, and GPU utilization. Validation should include the complete path from the sending GPU to the receiving GPU.

11. C-LIGHT Connectivity Products for AI Scale-Out Networks

Scale-out AI networks require high-speed connections between GPU servers, leaf switches, spine switches, and other data center equipment. C-LIGHT provides DAC, AEC, AOC, and optical transceiver products for supported high-speed connectivity applications.

12. Conclusion