What Is Scale-Up Networking in AI Data Centers?

1. What Is Scale-Up Networking?

Scale-up networking is an architecture that connects multiple processors or accelerators within a shared high-performance computing domain. In AI data centers, it primarily refers to the high-speed connections between GPUs or other AI accelerators inside a server, chassis, or rack-scale system.

Unlike conventional networking, which mainly connects independent servers, a scale-up fabric is designed to support frequent and intensive communication between accelerators. It provides the bandwidth and low latency needed for operations such as tensor transfers, gradient synchronization, parameter exchange, and distributed model execution.

The objective is to expand the computing resources available within a tightly connected system. Depending on the architecture, GPUs may communicate through direct links, switching fabrics, or accelerator-specific interconnect protocols.

2. Scale-Up, Scale-Out, and Scale-Across: What Is the Difference?

AI infrastructure uses multiple networking layers because different workloads require different communication distances and performance characteristics. Scale-Up, Scale-Out, and Scale-Across are related, but they do not describe the same type of connectivity.

These categories are complementary. A large AI data center may use a scale-up fabric inside each GPU rack, a scale-out network between racks, and additional networks for storage, management, and external connectivity.

3. Why Do AI Data Centers Need Scale-Up Networks?

Large language models, multimodal models, and mixture-of-experts architectures can require more accelerator memory and computing capacity than a single GPU can provide. Distributing a model across several GPUs allows the system to use their combined resources, but it also creates substantial communication requirements.

Expanding Available Computing Resources

Multiple GPUs can divide model computation and work on different portions of a task. Scale-up connectivity allows these accelerators to exchange intermediate results and coordinate their execution efficiently.

Supporting Large Model Memory Requirements

When a model is partitioned across GPUs, its parameters and intermediate tensors may be distributed across multiple memory spaces. High-speed GPU-to-GPU communication helps the system access and exchange the information required by each computation stage.

Reducing Communication Bottlenecks

In communication-intensive workloads, slow data transfers can leave GPUs waiting for information. A high-bandwidth, low-latency fabric helps reduce this waiting time and improve the utilization of expensive accelerator resources.

Improving Distributed Inference

Inference workloads can also benefit from scale-up architecture. Large models may be divided across GPUs, while mixture-of-experts models may route tokens between different expert groups. These operations can generate frequent accelerator-to-accelerator traffic.

4. How Does a Scale-Up Network Work?

A scale-up fabric connects accelerators through physical links and switching components. The exact implementation varies by platform, but the general architecture includes GPU endpoints, high-speed interconnect links, switches where required, and software that manages communication.

When one GPU needs data from another, the communication software selects an available path through the interconnect fabric. Depending on the platform, data may move over a direct GPU link or through one or more switching components.

Communication libraries coordinate transfers and collective operations. For NVIDIA GPU environments, NCCL is commonly used to manage operations such as All-Reduce, All-Gather, Reduce-Scatter, and Broadcast.

Some systems also support in-network computing, where compatible switches perform parts of collective operations instead of forwarding all data for processing at the GPU endpoints. This can reduce communication overhead for supported workloads.

5. Key Technologies Used in Scale-Up Networking

Several interconnect technologies are used or being developed for accelerator-scale systems. They differ in protocol, topology, ecosystem, and implementation.

NVLink

NVLink is NVIDIA's high-speed GPU interconnect technology. It provides direct communication paths between supported GPUs and other platform components, with higher bandwidth than relying exclusively on conventional PCIe connectivity in supported systems.

NVLink is designed for communication-intensive workloads, including distributed AI training and inference. Its bandwidth and topology vary by generation and platform.

NVSwitch

NVSwitch is a switching technology that expands GPU connectivity within supported NVIDIA systems. Rather than relying only on a limited number of direct GPU-to-GPU links, multiple accelerators can communicate through a high-bandwidth switching fabric.

In rack-scale platforms, NVSwitch-based systems can connect a larger group of GPUs into one scale-up domain. The number of GPUs and aggregate bandwidth depend on the specific generation and system design.

PCI Express (PCIe)

PCIe is a widely used interface connecting CPUs, GPUs, NICs, storage devices, and other components. It provides an important data path in AI servers and can support GPU peer-to-peer communication when the platform and topology permit it.

PCIe remains essential even in systems that use dedicated GPU interconnects. However, it does not automatically provide the same GPU-to-GPU topology or bandwidth characteristics as a specialized scale-up fabric.

UALink

UALink is an industry initiative focused on an open accelerator interconnect ecosystem. Its goal is to provide a standardized approach to connecting accelerators and supporting communication within scale-up systems.

UALink is relevant to the broader development of multi-vendor AI infrastructure. Its practical adoption depends on available silicon, accelerator platforms, switches, software support, and interoperable implementations.

Ethernet for Scale-Up Networking (ESUN)

Ethernet-based scale-up networking is another developing direction. The OCP Ethernet for Scale-Up Networking initiative is working toward open approaches for connecting accelerators through Ethernet-based fabrics.

The objective is to adapt Ethernet technology and related protocols to the demanding communication requirements of scale-up domains. This includes low-latency transport, traffic management, error handling, and reliable data delivery.

These technologies should not be treated as interchangeable products. Their suitability depends on the accelerator platform, network architecture, software ecosystem, and required communication behavior.

6. What Is Rack-Scale Computing?

Rack-scale computing extends the idea of a multi-GPU server to a larger integrated system. Instead of treating each server as an independent unit, the architecture connects multiple compute trays, accelerator modules, and switching components into a coordinated rack-level platform.

This design can provide a much larger scale-up domain than a conventional server. It also requires careful planning of power delivery, cooling, cabling, switch placement, serviceability, and system management.

Example: A 72-GPU Rack-Scale System

Some NVIDIA rack-scale systems use NVLink and NVSwitch to connect 72 GPUs within one rack-level scale-up domain. This allows the accelerators to communicate through a high-bandwidth fabric while operating as part of a larger coordinated system.

The same rack can also contain a separate scale-out network interface. That network connects the rack to other GPU racks through InfiniBand or Ethernet, extending the overall cluster beyond the local scale-up domain.

7. Scale-Up vs. Scale-Out: How the Two Networks Work Together

Scale-Up and Scale-Out are not competing alternatives. They form complementary layers in modern AI data center architecture.

Scale-Up provides dense communication among GPUs within a tightly connected domain. Scale-Out connects these domains across servers and racks, allowing the entire AI cluster to grow beyond the capacity of one rack.

For example, a large training workload may use several GPUs inside each rack for tensor parallelism and connect multiple racks through a scale-out fabric for broader distributed training.

The communication pattern determines which network layer carries the traffic. Frequent exchanges between GPUs in one rack may use the scale-up fabric, while communication between separate racks travels through the scale-out network.

8. The Role of Copper and Optical Interconnects

Scale-up networking places demanding requirements on physical connectivity. High-speed signals must travel between accelerators and switching components with acceptable signal integrity, latency, power consumption, and reliability.

For short electrical paths inside servers and rack-scale systems, copper interconnects remain important. As link distances increase or port density becomes more demanding, optical technologies become increasingly relevant.

DAC and Active Electrical Connectivity

Direct Attach Copper (DAC) cables provide short-reach electrical connections between compatible devices. They are commonly used for connections where cable length, signal integrity, and port compatibility meet the system requirements.

Active Electrical Cables (AEC) integrate active signal-conditioning components to support certain longer or more challenging electrical connections. Their use depends on the host interface, cable design, and supported application.

AOC and Optical Transceivers

Active Optical Cables (AOC) use optical fibers with integrated optical components at the cable ends. They can provide high-speed connectivity over supported distances while offering different routing and cable-management characteristics from copper.

Pluggable optical transceivers provide another option for optical links between network devices. They are particularly important in scale-out networks connecting GPU servers and switches across racks or other data center locations.

In some emerging rack-scale architectures, optical connectivity may also become part of the scale-up fabric as systems expand beyond the reach or practical density of conventional electrical interconnects. Co-packaged optics and other optical I/O approaches are being developed to address future bandwidth and power challenges.

9. What Are the Main Challenges of Scale-Up Networking?

Building a larger scale-up domain requires more than adding GPUs and connecting additional cables. The interconnect fabric must maintain predictable performance as the number of accelerators and communication paths increases.

Bandwidth Scaling

As more GPUs participate in a workload, the total communication demand can increase significantly. The interconnect fabric must provide enough aggregate bandwidth to avoid restricting accelerator utilization.

Latency and Synchronization

Collective operations often require multiple GPUs to exchange information and synchronize. Delays in one part of the fabric can affect the progress of the entire operation.

Topology Complexity

GPU count, link arrangement, switch stages, and routing determine the available communication paths. A poorly balanced topology can create bottlenecks even when individual links provide high bandwidth.

Power and Thermal Management

Dense accelerator systems consume substantial power. GPUs, switches, cables, and optical components all contribute to the thermal and power requirements of the platform. Cooling and power delivery must be designed together with the interconnect architecture.

Reliability and Serviceability

A rack-scale system contains many interconnected components. Link failures, switch issues, or maintenance operations can affect a large number of accelerators. Monitoring, fault isolation, redundancy, and service procedures are therefore important design considerations.

Interoperability

Different accelerator platforms may use different interconnect protocols and software stacks. Open standards and interoperable components can broaden the ecosystem, but practical compatibility must be verified across endpoints, switches, cables, and software.

10. How Scale-Up Networking Is Evolving

The development of AI models is pushing scale-up architectures beyond traditional multi-GPU servers. Rack-scale systems are increasing accelerator density, while industry initiatives are exploring more open approaches to connecting processors and switching fabrics.

Several technology directions are shaping this development:

• Higher-bandwidth GPU fabrics: New generations of accelerator interconnects aim to increase bandwidth and support larger communication domains.

• Rack-scale architectures: More accelerators are being integrated into coordinated systems with dedicated switching fabrics.

• Open interconnect ecosystems: Initiatives such as UALink and OCP ESUN are working toward broader interoperability and alternative scale-up approaches.

• Ethernet-based scale-up: Ethernet technologies are being adapted for accelerator communication within tightly connected domains.

• Optical I/O and co-packaged optics: Optical technologies may help address future bandwidth density, reach, and power challenges as electrical connectivity becomes more difficult to scale.

These approaches are at different stages of development and adoption. A technology initiative or published specification does not mean that every component is already available or interchangeable in production systems.

11. C-LIGHT Connectivity Products for AI Data Center Infrastructure

Although specialized GPU fabrics handle many intra-rack scale-up connections, AI data centers also require high-speed connectivity between servers, switches, racks, and network layers. These connections support the scale-out infrastructure surrounding the scale-up domain.

C-LIGHT offers high-speed connectivity products for data center and AI infrastructure applications, including:

12. Conclusion