Ethernet is widely used in enterprise networks, cloud infrastructure, and modern data centers. As artificial intelligence (AI), high-performance computing (HPC), and distributed GPU clusters continue to grow, conventional Ethernet communication is facing increasingly demanding requirements for bandwidth, latency, and CPU efficiency.
RoCE (RDMA over Converged Ethernet) addresses these requirements by enabling Remote Direct Memory Access (RDMA) over Ethernet networks. It allows servers to exchange data with reduced CPU involvement and lower communication overhead compared with conventional TCP/IP networking.
Although both RoCE and traditional Ethernet use Ethernet connectivity, they differ in their communication protocols, data transfer mechanisms, transport processing, and network configuration requirements. Understanding these differences is important when designing high-performance data center and AI networking infrastructure.
1. What Is RoCE?
RoCE stands for RDMA over Converged Ethernet. It is a networking technology that enables Remote Direct Memory Access over Ethernet infrastructure.
RDMA allows one computer to access memory on another computer without requiring the remote CPU to process every data transfer operation. This reduces CPU overhead and can improve communication efficiency in applications that exchange large amounts of data between servers.
RoCE is particularly relevant to environments where communication performance directly affects application efficiency, including:
AI training and GPU clusters
High-performance computing (HPC)
Distributed databases
High-performance storage systems
Cloud computing infrastructure
Scientific computing and simulation
Low-latency data processing
RoCE uses Ethernet as its underlying network technology. It does not replace Ethernet switching or physical connectivity. Instead, it provides an RDMA communication mechanism that operates over an Ethernet network.
2. What Is Traditional Ethernet?
Traditional Ethernet refers to conventional Ethernet communication, commonly using the TCP/IP protocol stack to transfer data between networked devices.
In a typical TCP/IP Ethernet environment, applications send data through the operating system's networking stack. TCP handles connection management, sequencing, acknowledgments, retransmission, and flow control to provide reliable data delivery.
Traditional Ethernet is widely used for:
Enterprise applications
Web servers and application servers
Database communication
Virtual machines and container networks
File sharing and general storage access
Cloud platforms
Internet and client-server communication
Its broad compatibility and mature ecosystem make traditional Ethernet suitable for a wide range of networking requirements. For most general-purpose applications, conventional TCP/IP communication provides sufficient performance and reliability.
3. RoCE vs Traditional Ethernet: Key Differences
| Feature | RoCE | Traditional Ethernet |
|---|---|---|
| Technology | RDMA over Ethernet | Conventional Ethernet communication |
| Communication model | Remote Direct Memory Access | Traditional socket-based communication |
| Common transport | RoCEv1 or RoCEv2 | Commonly TCP/IP |
| CPU involvement | Reduced through RDMA hardware offload | Generally involves the operating system networking stack |
| Data transfer | Direct memory operations through RDMA | Data handled through conventional networking paths |
| Latency | Designed for low-latency communication | Depends on protocol, hardware, and network conditions |
| CPU overhead | Typically lower for supported RDMA workloads | Generally higher for comparable data-transfer workloads |
| Congestion management | Often uses ECN and carefully configured network controls | Commonly relies on TCP congestion control and other network mechanisms |
| Network adapters | RDMA-capable Ethernet NICs | Standard Ethernet NICs |
| Switches | Compatible Ethernet switches with suitable configuration | Conventional Ethernet switches |
| Applications | AI, HPC, and high-performance storage | Enterprise, cloud, and general networking |
The key difference is the data communication mechanism. RoCE enables RDMA operations over Ethernet, while traditional Ethernet applications commonly use TCP/IP communication through the operating system networking stack.
4. How Does RoCE Work?
RoCE enables RDMA communication between systems connected through Ethernet networks. Instead of relying entirely on conventional TCP/IP socket processing, compatible RDMA network adapters handle memory-transfer operations directly.
A simplified RoCE communication process includes:
Memory registration: The application registers memory regions so that the RDMA network adapter can access them.
Queue setup: The communicating systems establish the required RDMA queue pairs and communication resources.
RDMA operation: The local network adapter processes an RDMA operation, such as a read, write, or send.
Network transmission: RDMA traffic is transmitted across the Ethernet network using the appropriate RoCE transport.
Remote memory access: The receiving adapter processes the operation and accesses the registered memory region according to the established permissions.
Completion notification: The application receives completion information through the RDMA completion mechanism.
This architecture reduces the amount of CPU processing required for supported data-transfer operations. It is especially useful when applications need to exchange large volumes of data between computing nodes.
5. RoCEv1 vs RoCEv2
RoCE has two major versions: RoCEv1 and RoCEv2. Both provide RDMA communication over Ethernet, but they differ in how packets are transported across the network.
| Feature | RoCEv1 | RoCEv2 |
|---|---|---|
| Transport | Ethernet Layer 2 | UDP/IP |
| Network scope | Same Layer 2 Ethernet domain | Can operate across routed IP networks |
| IP routing | Not supported natively | Supported |
| UDP destination port | Not applicable | 4791 |
| Deployment flexibility | More limited | Greater flexibility for multi-subnet environments |
| Common applications | Layer 2 RDMA networks | Modern data center and AI networks |
5.1 RoCEv1
RoCEv1 operates directly over Ethernet Layer 2. It uses an Ethernet-specific transport and requires communicating endpoints to be within the same Layer 2 domain.
This makes RoCEv1 suitable for environments where the RDMA network is designed around a single Layer 2 Ethernet domain. However, its limited routing capability can make large or geographically distributed network deployments more difficult.
5.2 RoCEv2
RoCEv2 encapsulates RDMA transport packets over UDP/IP. This allows RoCEv2 traffic to use IP routing and operate across different Layer 3 network segments.
RoCEv2 uses UDP destination port 4791 and is commonly used in modern Ethernet-based data centers and AI infrastructure.
Its use of IP routing provides greater network design flexibility, although high-performance RoCEv2 deployments still require careful consideration of routing, congestion control, packet loss, and traffic distribution.
6. RDMA vs Traditional TCP/IP Communication
The most important distinction between RoCE and traditional Ethernet communication is the role of RDMA.
In a conventional TCP/IP environment, data normally passes through the application, operating system networking stack, protocol processing, and network adapter. The receiving system also processes incoming data through its networking stack.
With RDMA, supported operations are handled by RDMA-capable network adapters, allowing data to be transferred directly between registered memory regions with reduced host CPU involvement.
| Communication Stage | Traditional TCP/IP | RoCE with RDMA |
|---|---|---|
| Application interface | Socket API | RDMA verbs or supported RDMA frameworks |
| Memory handling | Conventional application and kernel memory processing | Registered memory regions |
| Transport processing | TCP or UDP through the networking stack | RDMA transport handled by the RDMA-capable adapter |
| CPU involvement | Required for conventional networking processing | Reduced for supported RDMA operations |
| Data movement | Conventional networking data path | Direct memory transfer through RDMA hardware |
| Typical use | General application communication | High-performance data exchange |
RDMA does not eliminate all CPU activity. Applications still need to establish connections, register memory, manage queues, and process completion events. The advantage is that the data-transfer path can avoid much of the conventional host networking overhead.
7. Latency and CPU Utilization
Latency and CPU utilization are two major reasons why organizations deploy RoCE in high-performance networking environments.
7.1 Latency
Conventional TCP/IP communication includes protocol processing, connection management, acknowledgments, and other operations that can contribute to end-to-end latency.
RoCE reduces communication overhead by allowing RDMA-capable network adapters to handle supported data-transfer operations. This can provide lower latency and more predictable performance in appropriately configured environments.
However, actual latency depends on network adapter capabilities, switch forwarding performance, network topology, congestion, packet size, and application behavior.
7.2 CPU Utilization
Traditional TCP/IP networking requires CPU resources for tasks such as protocol processing, packet handling, and memory management.
RoCE offloads supported RDMA operations to the network adapter, reducing the CPU workload associated with data movement.
This is particularly valuable in GPU servers, where CPU resources are also required for application coordination, storage operations, and system management.
8. Congestion Control in RoCE Networks
Congestion management is one of the most important considerations when deploying RoCE over Ethernet.
Traditional TCP uses congestion control mechanisms that respond to packet loss and other network feedback. RoCEv2 does not use TCP for its RDMA transport, so a RoCE deployment requires suitable congestion-management mechanisms to maintain reliable and predictable communication.
Common technologies include:
ECN (Explicit Congestion Notification): Network switches can mark packets when congestion occurs, allowing endpoints to adjust their transmission rates.
CNP (Congestion Notification Packet): In RoCEv2 congestion-control implementations, a receiver can send a notification to the sender when congestion is indicated.
PFC (Priority Flow Control): Provides priority-based pause mechanisms to help prevent packet loss for selected traffic classes. It requires careful configuration to avoid congestion spreading through the network.
Traffic Prioritization: Helps separate and manage different traffic classes according to application requirements.
Adaptive Routing: Distributes traffic across available network paths where supported.
Not every RoCE deployment uses exactly the same congestion-control configuration. The appropriate design depends on network equipment, NIC capabilities, software support, and workload requirements.
9. Network Requirements for RoCE
RoCE can operate over Ethernet infrastructure, but it requires compatible hardware and appropriate network configuration.
Important deployment requirements include:
RDMA-capable NICs: Network adapters must support the required RoCE version and RDMA operations.
Compatible Ethernet switches: Switches should support the required bandwidth, traffic handling, and congestion-management features.
Appropriate network configuration: VLANs, QoS, ECN, and PFC settings should be planned according to the deployment design.
Low-congestion topology: Network paths should provide sufficient capacity for expected traffic patterns.
Compatible drivers and firmware: NIC drivers, firmware, and RDMA software must work together.
High-speed connectivity: Optical transceivers, DAC, and AOC cables should match the port speed and transmission distance.
Simply enabling RoCE on an Ethernet NIC does not automatically create a high-performance RDMA network. The complete system must be configured and validated.
10. RoCE vs Traditional Ethernet: Bandwidth and Interface Speeds
RoCE does not define a separate Ethernet port speed. It operates over supported Ethernet interfaces, meaning the available bandwidth depends on the underlying network adapter, switch, and physical link.
Common Ethernet speeds used in RoCE deployments include:
| Ethernet Speed | Potential Application | Network Consideration |
|---|---|---|
| 25G | Smaller server and storage deployments | Lower-bandwidth RDMA connectivity |
| 100G | Data center and storage networks | High-speed server communication |
| 200G | High-performance computing | Higher-bandwidth RDMA connections |
| 400G | AI clusters and modern data centers | High-capacity RoCE fabrics |
| 800G | Large-scale AI infrastructure | High-density next-generation connectivity |
| 1.6T | Emerging data center infrastructure | Future high-bandwidth networking |
The choice of interface speed should be based on server capabilities, network oversubscription, expected traffic volume, and the required performance of the application.
11. Optical Transceivers for RoCE Networks
Optical transceivers are important in high-speed RoCE networks, especially when connecting servers to leaf switches or linking leaf and spine switches across a data center.
Common optical connectivity options include:
100G QSFP28 optical transceivers
200G QSFP56 optical transceivers
400G QSFP-DD optical transceivers
400G QSFP112 optical transceivers
800G QSFP-DD optical transceivers
800G OSFP optical transceivers
For 400G RoCE infrastructure, C-LIGHT 400G QSFP-DD optical transceivers provide multiple optical configurations for data center connectivity. Available options include SR8, DR4, FR4, and LR4 variants, with different fiber types and transmission distances.
For higher-bandwidth Ethernet infrastructure, C-LIGHT 800G QSFP-DD optical transceivers support next-generation data center and AI networking applications. The product family includes 100m, 500m, and 2km transmission options, depending on the specific model.
When selecting optical modules for RoCE networks, the physical form factor alone is not sufficient. The Ethernet protocol, host interface, switch compatibility, optical reach, fiber type, and supported data rate must all be verified.
12. DAC and AOC Cables for RoCE Connectivity
In addition to pluggable optical transceivers, DAC and AOC cables are widely used for short-distance Ethernet interconnections.
12.1 DAC Cables
Direct Attach Copper (DAC) cables use integrated electrical connections to link compatible network ports. They are commonly used for short-distance connections between servers and switches, particularly within the same rack.
DAC cables can provide a practical option where transmission distance is limited and the equipment supports the required electrical interface.
12.2 AOC Cables
Active Optical Cables (AOCs) integrate optical transmitters and receivers into a cable assembly. They provide optical transmission over short distances without requiring separate pluggable optical transceivers and fiber patch cables.
For 400G RoCE applications, C-LIGHT 400G AOC cables include configurations identified for 400G InfiniBand and Ethernet-related connectivity. The exact product configuration should be selected according to the target protocol and equipment compatibility.
For next-generation 800G environments, C-LIGHT 800G AOC cables include both InfiniBand and Ethernet configurations. The product family supports short-reach connections using QSFP-DD and OSFP interfaces, with different breakout options.
It is important to distinguish Ethernet and InfiniBand configurations when selecting an AOC. A similar connector or form factor does not guarantee protocol compatibility.
13. RoCE vs Traditional Ethernet: Application Scenarios
| Application | RoCE | Traditional Ethernet |
|---|---|---|
| AI model training | Highly suitable for RDMA-based communication | Suitable depending on workload and performance requirements |
| GPU cluster communication | Suitable for high-speed data exchange | Commonly used for general server communication |
| High-performance storage | Suitable for supported RDMA storage protocols | Suitable for conventional TCP/IP storage access |
| Database applications | Useful for selected high-performance database workloads | Widely used |
| Enterprise applications | Usually unnecessary for general workloads | Widely used |
| Cloud computing | Suitable for specialized high-performance infrastructure | Common general-purpose networking foundation |
| Scientific computing | Suitable for RDMA-intensive applications | Suitable for general communication requirements |
| Virtualization | Possible with compatible platforms and configurations | Common |
RoCE is most valuable when applications can take advantage of RDMA. For general-purpose enterprise communication, traditional Ethernet is often simpler and sufficient.
14. When Should You Use RoCE Instead of Traditional Ethernet?
The decision depends on application requirements rather than network speed alone.
Choose RoCE When:
Your application requires low-latency communication between servers.
Large volumes of data must be transferred with reduced CPU overhead.
You are building an AI training or GPU computing cluster.
Your storage platform supports RDMA-based data access.
You have RDMA-capable NICs and compatible Ethernet switches.
Your network team can configure and monitor the required congestion-management mechanisms.
Choose Traditional Ethernet When:
You are deploying general enterprise applications.
Your workloads primarily use conventional TCP/IP communication.
Application compatibility and operational simplicity are priorities.
RDMA capabilities are not required.
Your existing Ethernet infrastructure already meets application performance requirements.
For some data centers, both technologies can coexist. General application traffic can use conventional TCP/IP Ethernet, while selected high-performance workloads use RoCE over dedicated or logically separated Ethernet resources.
15. C-LIGHT Optical Connectivity for RoCE Networks
As AI and high-performance computing workloads increase network bandwidth requirements, optical connectivity becomes an important part of RoCE infrastructure.
C-LIGHT provides optical transceivers and high-speed interconnect solutions for data centers, AI networks, and high-performance computing environments.
400G QSFP-DD Optical Transceivers for high-speed Ethernet and data center connectivity.
800G QSFP-DD Optical Transceivers for next-generation AI and high-bandwidth Ethernet infrastructure.
400G AOC Cables for short-reach high-speed interconnections, including specified InfiniBand configurations.
800G AOC Cables for high-density data center and high-performance computing connections.
C-LIGHT Optical Connectivity Products for additional data rates, form factors, and transmission distances.
When building a RoCE network, selecting the correct optical connectivity requires checking the Ethernet interface speed, NIC and switch compatibility, transmission distance, fiber type, and cable configuration.
16. Frequently Asked Questions
Q1: What is the main difference between RoCE and Traditional Ethernet?
Q2: Is RoCE a separate Ethernet standard?
Q3: Does RoCE use TCP?
Q4: What is the difference between RoCEv1 and RoCEv2?
Q5: Does RoCE reduce CPU utilization?
Q6: Does RoCE require PFC?
Q7: Can RoCE work with standard Ethernet switches?
Q8: Is RoCE suitable for AI data centers?
Q9: What optical transceivers are used in RoCE networks?
Q10: Can RoCE and Traditional Ethernet operate in the same data center?
17. Conclusion
RoCE and traditional Ethernet share the same Ethernet foundation, but they use different communication mechanisms. Traditional Ethernet commonly relies on TCP/IP to provide general-purpose network connectivity, while RoCE enables RDMA operations over Ethernet to reduce CPU overhead and improve data-transfer efficiency.
RoCE is particularly valuable for AI training, GPU clusters, high-performance storage, and other workloads that require low-latency communication and efficient data movement. However, it requires RDMA-capable network adapters, compatible switches, and carefully designed congestion management.
As data center networks continue to evolve toward 400G, 800G, and 1.6T connectivity, high-speed optical transceivers, DAC cables, and AOCs will remain important components of modern Ethernet infrastructure. Choosing the right connectivity solution helps ensure that the physical network can support the performance requirements of both conventional Ethernet and RDMA-based RoCE applications.
TEL:+86 132 6656 7067




















































>
>
>
>
>
>
>
>