
AI infrastructure is expanding in two directions at the same time. Individual accelerator systems are becoming more tightly interconnected, while data centers are connecting increasingly large numbers of GPU servers into distributed computing clusters. These two growth paths are known as AI scale-up and AI scale-out.
Scale-up increases the communication capacity among accelerators within a closely coupled computing domain. Scale-out extends the system by connecting multiple compute nodes through a high-performance network fabric. Both are essential to large AI training and inference deployments, but they solve different communication and capacity problems.
The choice of interconnect technology, network topology, optical connectivity, and cooling strategy depends on where the communication takes place and how the workload distributes computation across accelerators.
1. What Is AI Scale-Up Architecture?
AI scale-up architecture connects multiple processors or accelerators into a tightly coupled computing system. Its purpose is to make communication between GPUs efficient enough that they can cooperate on a workload with minimal communication overhead.
Instead of treating each GPU as an isolated computing device, a scale-up domain allows accelerators to exchange data and access shared or distributed memory resources through high-bandwidth interconnects. This is particularly important when a model is too large for one accelerator or when tensor operations need to be distributed across several GPUs.
Typical scale-up technologies include:
NVLink and NVSwitch: NVIDIA interconnect technologies used to create high-bandwidth GPU communication domains.
AMD Infinity Fabric and XGMI: Technologies used for communication among AMD processors and accelerators.
PCI Express: A widely used host and device interconnect that also provides connectivity within accelerator systems.
Other vendor-specific accelerator fabrics: Proprietary or standards-based interconnects designed for tightly coupled accelerator communication.
The physical scope of scale-up varies by platform. It may cover multiple GPUs within one server, or extend across several systems in a rack-scale architecture. NVIDIA GB200 NVL72, for example, uses NVLink and NVLink switches to connect GPUs across a rack-scale system.
Typical scale-up communication domain
Conceptual illustration: actual systems may use switched, mesh, or other accelerator interconnect topologies.
Scale-up performance depends on more than the peak bandwidth of an individual link. GPU-to-GPU communication patterns, switch architecture, memory access behavior, topology, and software scheduling all influence how efficiently accelerators work together.
2. What Is AI Scale-Out Architecture?
AI scale-out architecture expands computing capacity by connecting multiple GPU servers or accelerator systems through a distributed network. Each server contains its own processors, memory, and local interconnects, while the scale-out fabric enables communication between different nodes.
This architecture supports distributed training, large inference deployments, and workloads that require more accelerators than can be accommodated within one scale-up domain.
Common scale-out network technologies include:
InfiniBand: A high-performance networking technology widely deployed in AI and high-performance computing clusters.
High-performance Ethernet: Ethernet-based fabrics designed with congestion management, traffic control, and low-latency communication capabilities.
Ultra Ethernet: An industry initiative developing an Ethernet-based communications architecture for AI and HPC workloads.
Optical interconnects: Pluggable transceivers and optical cable solutions that provide connectivity between network devices and compute systems over supported distances.
Typical scale-out cluster architecture
Conceptual illustration: the scale-out fabric connects separate compute nodes and supporting infrastructure. Actual network designs vary by deployment.
Scale-out networks commonly use high-radix switches and multi-tier topologies such as leaf-spine or fat-tree designs. These structures provide paths between servers distributed across racks and help the cluster grow without requiring every accelerator to connect directly to every other accelerator.
As the cluster grows, network design must account for oversubscription, path diversity, congestion, collective communication, and failure recovery. Adding more GPU servers does not automatically produce proportional application performance if the network becomes a bottleneck.
3. AI Scale-Up vs AI Scale-Out: Key Technical Differences
Although both architectures increase the effective computing capacity of AI infrastructure, they operate at different levels of the system. Scale-up focuses on communication within a tightly coupled accelerator domain. Scale-out focuses on communication between independent compute nodes.
| Technical Dimension | AI Scale-Up | AI Scale-Out |
|---|---|---|
| Primary purpose | Increase the capacity and coordination of a tightly coupled accelerator system | Expand computing resources across multiple servers and systems |
| Communication scope | Within a server or rack-scale accelerator domain, depending on platform | Between servers, racks, and larger cluster sections |
| Typical endpoints | GPUs, CPUs, and accelerator switches | GPU servers, NICs, network switches, and storage systems |
| Common technologies | NVLink, NVSwitch, XGMI, PCIe, and vendor-specific accelerator fabrics | InfiniBand, high-performance Ethernet, and optical network links |
| Network characteristics | Very high local bandwidth and tightly integrated communication | Cluster-wide bandwidth, scalable routing, congestion control, and path redundancy |
| Physical connectivity | Short-reach electrical or optical connections, depending on system design | Electrical and optical links across servers, switches, and racks |
| Expansion method | Increase the number or capability of accelerators in a scale-up domain | Add more compute nodes, network capacity, and supporting infrastructure |
| Key design concerns | GPU bandwidth, topology, memory access, power, and thermal density | Network topology, oversubscription, congestion, cabling, and cluster management |
| Typical workload role | Fast communication among accelerators cooperating on the same workload | Distributed training and inference across multiple compute nodes |
These categories describe architectural roles rather than fixed physical standards. The technologies used for scale-up and scale-out can evolve, and some newer networking initiatives are designed to address requirements across both domains.
4. Scale-Up and Scale-Out in Distributed AI Training
Large AI training workloads often use both architectures in the same computing environment. A model may be distributed across several GPUs within a server or rack-scale system, while additional groups of accelerators participate through the cluster network.
For example, tensor parallelism can divide model operations across closely connected GPUs. Pipeline parallelism can distribute different model stages across devices, while data parallelism can replicate model parameters across multiple groups of accelerators. These methods generate different communication patterns and place different demands on the interconnect.
A simplified training workflow may look like this:
Local accelerator communication: GPUs exchange intermediate tensors, gradients, or other data through the scale-up fabric.
Cross-node communication: GPU servers exchange distributed training data through the scale-out network.
Synchronization: Collective operations such as all-reduce or all-gather coordinate data across participating accelerators.
Checkpointing: Training state is transferred to storage through the relevant storage and data network paths.
The balance between scale-up and scale-out depends on model size, parallelism strategy, accelerator count, communication volume, and the performance of the software stack.
5. Network Fabric and Optical Connectivity Requirements
Scale-up and scale-out architectures place different requirements on physical connectivity. Scale-up systems often use dedicated short-reach links and specialized switching hardware to connect accelerators within a compact domain. Scale-out networks must connect a much larger number of endpoints across servers and racks, making optical connectivity increasingly important as link rates and physical distances increase.
Scale-up connectivity
Within a tightly integrated accelerator system, the connection may use high-speed electrical traces, copper cables, backplane connections, or optical links, depending on the platform. The key requirements include signal integrity, low communication overhead, high aggregate bandwidth, and reliable operation under concentrated power and thermal loads.
Scale-out connectivity
Between compute nodes and network switches, the appropriate medium depends on link speed, reach, port density, and system design. Common options include:
DAC: Direct Attach Copper cables are suitable for supported short-reach connections, often within a rack.
AEC: Active Electrical Cables support selected high-speed electrical connections where signal conditioning is required.
AOC: Active Optical Cables provide optical transmission in an integrated cable assembly.
Pluggable optical transceivers: Modules such as 400G, 800G, and 1.6T solutions support optical links between compatible network interfaces.
Fiber-optic cabling: Single-mode or multimode fiber is selected according to the optical interface, reach, and deployment environment.
For AI scale-out networks, the move toward 400G, 800G, and 1.6T interfaces increases the importance of transceiver power, thermal design, fiber routing, connector density, and host-platform compatibility. The nominal port rate alone does not determine the actual throughput of a distributed training workload.
Relevant connectivity options include optical transceivers and the broader C-LIGHT product portfolio. Product selection should be based on the actual switch or NIC model, supported interface standard, optical reach, wavelength, fiber type, connector, and link budget.
6. Power, Cooling, and Physical Infrastructure
Scale-up systems concentrate many high-performance accelerators and interconnect components in a relatively small physical space. This creates demanding requirements for power delivery, heat removal, and rack-level mechanical design. Scale-out deployments distribute resources across more servers and switches, increasing the total infrastructure footprint and the number of network connections that must be managed.
| Infrastructure Area | Scale-Up Considerations | Scale-Out Considerations |
|---|---|---|
| Power delivery | High power concentration around accelerators and local interconnect hardware | Power distribution across compute racks, network racks, and supporting systems |
| Cooling | High thermal density and possible direct liquid cooling requirements | Cooling for a larger population of servers, switches, and optical modules |
| Cabling | Dense local interconnects and careful cable or backplane management | Large numbers of server-to-switch and switch-to-switch connections |
| Physical space | Rack-scale integration and high-density equipment layout | Additional racks, network tiers, fiber pathways, and service access |
| Maintenance | Platform-level component and system service planning | Network path redundancy, link monitoring, and distributed fault isolation |
Optical transceiver thermal performance is especially relevant in high-density switch systems. As port speeds increase, module power and heat dissipation must be considered alongside switch airflow, front-panel spacing, and the overall cooling strategy.
7. How to Choose the Right Architecture for an AI Data Center
Scale-up and scale-out are complementary design choices, not competing alternatives. The most suitable balance depends on the size of the model, the number of accelerators, the communication pattern, and the required expansion path.
For tightly coupled GPU workloads: Prioritize a scale-up fabric that provides sufficient accelerator-to-accelerator bandwidth and supports the required memory and communication model.
For large distributed training clusters: Design a scale-out network with sufficient bandwidth, predictable latency, congestion management, and resilient paths between compute nodes.
For multi-rack AI deployments: Evaluate optical links, switch radix, link reach, fiber management, and network oversubscription as part of the cluster architecture.
For growing infrastructure: Plan how additional accelerator domains will connect to the existing fabric, including switch capacity, port availability, power, cooling, and cabling pathways.
For mixed AI and storage traffic: Define traffic isolation, quality-of-service policies, and bandwidth allocation so that data access and checkpoint operations do not unnecessarily interfere with GPU communication.
Before deploying high-speed optical connections, verify the complete link specification. The transceiver's form factor, electrical interface, Ethernet or InfiniBand application, wavelength, fiber type, connector, transmission distance, FEC requirements, and host compatibility must all match the intended system.
8. Conclusion
AI scale-up and AI scale-out address different dimensions of accelerator infrastructure. Scale-up creates a high-bandwidth communication domain among closely connected GPUs, while scale-out connects multiple compute systems into a larger distributed cluster. A capable AI data center needs both local accelerator communication and scalable inter-node networking.
As AI clusters grow, the performance of the overall system increasingly depends on the relationship between accelerator interconnects, network fabrics, optical connectivity, power delivery, and cooling. Selecting the right combination of electrical and optical links helps build a network that can support current workloads while leaving room for future expansion.
For AI data center connectivity projects, C-LIGHT provides optical transceiver and connectivity solutions for different network speeds and deployment requirements. Contact our team to discuss interface compatibility, link reach, and product selection. Contact C-LIGHT.
TEL:+86 132 6656 7067




















































>
>
>
>
>
>
>
>