How Do Scale-Up and Scale-Out Networks Work Together?

1. Why AI Infrastructure Needs Both Scale-Up and Scale-Out

A single GPU has limited memory capacity and computing resources. Large language models, multimodal AI systems, and other advanced workloads often require multiple GPUs to work together. As model size and training requirements increase, a single server may no longer provide sufficient accelerator capacity.

Two networking dimensions help address this challenge:

• Scale-Up networking connects GPUs within a tightly coupled computing domain, providing high bandwidth and low-latency communication.

• Scale-Out networking connects multiple GPU servers and racks, expanding the available computing resources across a larger cluster.

These networks perform different but complementary functions. Scale-Up makes communication between nearby GPUs efficient, while Scale-Out allows the workload to extend beyond the limits of one server or rack.

2. What Does Each Network Layer Do?

Scale-Up: Connecting GPUs into a High-Bandwidth Domain

Scale-Up networking creates fast communication paths between multiple GPUs. Depending on the platform, this connection may operate within a server, across a multi-GPU system, or within a rack-scale architecture.

Common technologies include:

• NVIDIA NVLink: A high-speed GPU interconnect that supports direct GPU communication and memory access within supported systems.

• NVIDIA NVSwitch: A switching technology that connects multiple NVLink-enabled GPUs and provides scalable GPU-to-GPU communication within supported systems.

• AMD Infinity Fabric and XGMI: Technologies used for high-bandwidth communication between AMD processors and accelerators.

• UALink: An industry initiative defining an open accelerator interconnect architecture for scale-up systems.

Scale-Up is especially important for model parallelism, where a model is distributed across multiple GPUs. These GPUs frequently exchange intermediate tensors, activations, and other data during computation.

Scale-Out: Connecting Multiple GPU Servers

Scale-Out networking connects GPU servers across racks and network switches. It allows a workload to use more GPUs than can fit within a single tightly coupled system.

Common technologies include:

• InfiniBand: A high-performance networking technology widely used in HPC and large AI clusters.

• RoCEv2: RDMA over Converged Ethernet, enabling low-overhead data transfers over Ethernet networks.

• High-speed Ethernet: Provides the switching and routing infrastructure required to connect large numbers of compute nodes.

• AI NICs and DPUs: Support network communication, data movement, traffic management, and infrastructure acceleration.

Scale-Out is important for data parallelism and other distributed computing methods, where multiple servers process different portions of a workload and exchange results through the network.

3. How the Two Networks Connect in an AI Cluster

A typical multi-server AI cluster contains GPUs connected through a local Scale-Up fabric. Each server also has network interfaces that connect its GPU resources to the external Scale-Out fabric.

The following simplified structure illustrates how the two layers work together:

In this architecture, Scale-Up handles communication between GPUs inside each computing domain. The GPU NICs provide access to the Scale-Out network, which carries traffic between servers.

The front-end and management networks support the surrounding infrastructure. They may use separate physical or logical networks to isolate application traffic, storage access, and system administration from the GPU back-end fabric.

4. What Happens When a Workload Uses Both Networks?

Consider an AI training job distributed across several GPU servers. Each server contains multiple GPUs, and the training framework divides the model and data among them.

Communication takes place at two levels.

Step 1: GPUs Exchange Data Locally

Within each server or supported rack-scale domain, the Scale-Up fabric provides high-speed communication between GPUs. When a model is divided across GPUs, they exchange intermediate results needed to complete each stage of computation.

For example, tensor parallelism may require frequent exchanges of tensor data between GPUs. A high-bandwidth local interconnect helps reduce the communication overhead of these operations.

Step 2: Servers Exchange Data Across the Cluster

When the workload spans multiple servers, the Scale-Out fabric connects the GPU nodes. Network interfaces and RDMA-capable communication paths allow data to move between GPUs located on different servers.

For example, data-parallel training may require gradient synchronization across multiple nodes. The Scale-Out network carries this communication between servers.

Step 3: Collective Operations Coordinate the GPUs

Distributed AI frameworks use collective communication operations to coordinate computation across GPUs. Common operations include:

Communication libraries such as NVIDIA NCCL and AMD RCCL coordinate these operations across supported GPU systems. Depending on the topology, communication may use local GPU links, the external network, or a combination of both.

5. A Practical Example: Training Across Multiple GPU Servers

Imagine a training cluster with four GPU servers, each containing eight GPUs. The total system has 32 GPUs.

Within each server, the eight GPUs communicate through a local Scale-Up fabric. Across servers, the Scale-Out network connects the four nodes.

During training, each server performs computations using its local GPUs. The Scale-Up fabric handles communication among GPUs within that server. When information must be exchanged with other servers, the Scale-Out fabric carries the data across the cluster.

This layered arrangement allows the training framework to distribute computation while maintaining fast local GPU communication.

The example illustrates the networking concept, not a prescribed cluster design. Actual performance depends on GPU topology, NIC bandwidth, switch capacity, oversubscription, software configuration, and the workload's communication pattern.

6. How Scale-Up and Scale-Out Support Different AI Workloads

Different AI workloads place different demands on the two networking layers. The balance between local GPU communication and cross-server traffic depends on model architecture and parallelism strategy.

For example, tensor-parallel operations often involve frequent communication and can be sensitive to latency. Data-parallel workloads may generate substantial synchronization traffic between servers. Mixture-of-Experts models can create additional communication demands when tokens must be routed to experts distributed across different GPUs or nodes.

7. The Role of NICs, RDMA, and GPU Direct Data Transfer

The connection between Scale-Up and Scale-Out depends on more than switches and cables. Network interfaces and data-transfer mechanisms determine how efficiently GPU data reaches the external network.

GPU Network Interface Cards

High-performance NICs connect GPU servers to the Scale-Out fabric. They provide the network bandwidth required for communication between nodes and may include hardware acceleration for packet processing and RDMA operations.

RDMA and RoCEv2

Remote Direct Memory Access allows supported systems to transfer data between memory regions on different machines with reduced CPU involvement. RoCEv2 carries RDMA traffic over Ethernet networks.

InfiniBand also supports RDMA and is widely used in high-performance computing and AI clusters. Both technologies can support GPU-oriented data movement when the hardware, drivers, and software stack are configured appropriately.

GPUDirect RDMA

GPUDirect RDMA enables supported network adapters to access GPU memory directly, reducing unnecessary data copies through host memory. This can improve communication efficiency for workloads that exchange large amounts of data between GPU nodes.

However, the actual data path depends on the GPU platform, NIC, PCIe or other host interconnect topology, driver support, and software configuration.

8. Why Optical Connectivity Matters for Scale-Out

As GPU clusters expand across racks, the network must support increasing bandwidth and physical reach. Copper and optical connections both have important roles, but their suitability depends on distance, port speed, power, and deployment requirements.

DAC: Short-Reach Copper Connectivity

Direct Attach Copper cables provide an electrical connection between compatible ports. They are commonly used for short connections within a rack or between nearby devices where the cable reach and signal integrity meet the system requirements.

AEC: Active Electrical Connectivity

Active Electrical Cables incorporate electronic signal-conditioning functions to support selected high-speed connections. They can provide an alternative to passive DAC in deployments where additional signal conditioning is needed.

AOC: Active Optical Connectivity

Active Optical Cables use optical transmission within an integrated cable assembly. They are useful for supported connections where longer reach, cable flexibility, or electromagnetic isolation is important.

Pluggable Optical Transceivers

Optical transceivers provide detachable optical interfaces for switch-to-switch and switch-to-server connections. They support different data rates, distances, wavelengths, and fiber types.

In a large AI cluster, the choice of connectivity should follow the physical network layout. DAC may be suitable for short in-rack connections, while AOC and pluggable optics can support other distances and deployment arrangements.

9. Bandwidth Planning: Avoiding Bottlenecks Between the Two Layers

A high-performance Scale-Up fabric does not automatically guarantee high cluster performance. If the external Scale-Out network cannot carry data efficiently, GPUs may spend time waiting for communication to complete.

Several factors should be evaluated together:

• GPU-to-NIC bandwidth: The available bandwidth between GPU memory and the network interface.

• Network port speed: The aggregate bandwidth available to each server.

• Switch capacity: The ability of the switching fabric to handle simultaneous traffic flows.

• Oversubscription: The relationship between the total server-facing bandwidth and the available uplink bandwidth.

• Congestion control: How the network manages traffic bursts and competing flows.

• Topology: The number of network hops, path diversity, and communication pattern.

• Workload communication: How frequently GPUs exchange data and how much computation can overlap with communication.

10. Common Design Challenges

Unbalanced Network Capacity

If the Scale-Up domain has substantial aggregate bandwidth but the NIC and Scale-Out fabric provide insufficient external capacity, cross-server communication may become a bottleneck.

Latency-Sensitive Communication

Some distributed workloads perform frequent synchronization. Network latency and tail latency can influence how long GPUs wait for other participants to complete communication.

Congestion and Packet Loss

Large numbers of GPUs can generate synchronized traffic bursts. Poorly managed congestion may increase queueing delays, trigger retransmissions, and reduce effective throughput.

Topology and Traffic Placement

GPU placement, server grouping, and network topology should reflect the communication patterns of the workload. Keeping frequently communicating GPUs within an appropriate local domain can reduce unnecessary cross-network traffic.

Thermal and Power Constraints

High-speed switches, NICs, and optical modules contribute to rack-level power and thermal requirements. Network design must account for airflow, cooling capacity, module power, and equipment density.

11. C-LIGHT Connectivity Products for AI Scale-Out Networks

C-LIGHT provides high-speed connectivity products for data center networking, including DAC, AEC, AOC, and optical transceiver solutions. These products can support different connection layers in AI infrastructure, depending on the required distance, data rate, and host platform.

12. Conclusion