C-LIGHT telephone TEL:+86 132 6656 7067    
Language
C-LIGHT search

RoCE vs Traditional Ethernet

By C-LIGHT Marketing 丨 Jul 28, 2025
Table of Contents

    Ethernet is widely used in enterprise networks, cloud infrastructure, and modern data centers. As artificial intelligence (AI), high-performance computing (HPC), and distributed GPU clusters continue to grow, conventional Ethernet communication is facing increasingly demanding requirements for bandwidth, latency, and CPU efficiency.

    RoCE (RDMA over Converged Ethernet) addresses these requirements by enabling Remote Direct Memory Access (RDMA) over Ethernet networks. It allows servers to exchange data with reduced CPU involvement and lower communication overhead compared with conventional TCP/IP networking.

    Although both RoCE and traditional Ethernet use Ethernet connectivity, they differ in their communication protocols, data transfer mechanisms, transport processing, and network configuration requirements. Understanding these differences is important when designing high-performance data center and AI networking infrastructure.

    1. What Is RoCE?

    RoCE stands for RDMA over Converged Ethernet. It is a networking technology that enables Remote Direct Memory Access over Ethernet infrastructure.

    RDMA allows one computer to access memory on another computer without requiring the remote CPU to process every data transfer operation. This reduces CPU overhead and can improve communication efficiency in applications that exchange large amounts of data between servers.

    RoCE is particularly relevant to environments where communication performance directly affects application efficiency, including:

    • AI training and GPU clusters

    • High-performance computing (HPC)

    • Distributed databases

    • High-performance storage systems

    • Cloud computing infrastructure

    • Scientific computing and simulation

    • Low-latency data processing

    RoCE uses Ethernet as its underlying network technology. It does not replace Ethernet switching or physical connectivity. Instead, it provides an RDMA communication mechanism that operates over an Ethernet network.

    2. What Is Traditional Ethernet?

    Traditional Ethernet refers to conventional Ethernet communication, commonly using the TCP/IP protocol stack to transfer data between networked devices.

    In a typical TCP/IP Ethernet environment, applications send data through the operating system's networking stack. TCP handles connection management, sequencing, acknowledgments, retransmission, and flow control to provide reliable data delivery.

    Traditional Ethernet is widely used for:

    • Enterprise applications

    • Web servers and application servers

    • Database communication

    • Virtual machines and container networks

    • File sharing and general storage access

    • Cloud platforms

    • Internet and client-server communication

    Its broad compatibility and mature ecosystem make traditional Ethernet suitable for a wide range of networking requirements. For most general-purpose applications, conventional TCP/IP communication provides sufficient performance and reliability.

    3. RoCE vs Traditional Ethernet: Key Differences

    FeatureRoCETraditional Ethernet
    TechnologyRDMA over EthernetConventional Ethernet communication
    Communication modelRemote Direct Memory AccessTraditional socket-based communication
    Common transportRoCEv1 or RoCEv2Commonly TCP/IP
    CPU involvementReduced through RDMA hardware offloadGenerally involves the operating system networking stack
    Data transferDirect memory operations through RDMAData handled through conventional networking paths
    LatencyDesigned for low-latency communicationDepends on protocol, hardware, and network conditions
    CPU overheadTypically lower for supported RDMA workloadsGenerally higher for comparable data-transfer workloads
    Congestion managementOften uses ECN and carefully configured network controlsCommonly relies on TCP congestion control and other network mechanisms
    Network adaptersRDMA-capable Ethernet NICsStandard Ethernet NICs
    SwitchesCompatible Ethernet switches with suitable configurationConventional Ethernet switches
    ApplicationsAI, HPC, and high-performance storageEnterprise, cloud, and general networking

    The key difference is the data communication mechanism. RoCE enables RDMA operations over Ethernet, while traditional Ethernet applications commonly use TCP/IP communication through the operating system networking stack.

    4. How Does RoCE Work?

    RoCE enables RDMA communication between systems connected through Ethernet networks. Instead of relying entirely on conventional TCP/IP socket processing, compatible RDMA network adapters handle memory-transfer operations directly.

    A simplified RoCE communication process includes:

    1. Memory registration: The application registers memory regions so that the RDMA network adapter can access them.

    2. Queue setup: The communicating systems establish the required RDMA queue pairs and communication resources.

    3. RDMA operation: The local network adapter processes an RDMA operation, such as a read, write, or send.

    4. Network transmission: RDMA traffic is transmitted across the Ethernet network using the appropriate RoCE transport.

    5. Remote memory access: The receiving adapter processes the operation and accesses the registered memory region according to the established permissions.

    6. Completion notification: The application receives completion information through the RDMA completion mechanism.

    This architecture reduces the amount of CPU processing required for supported data-transfer operations. It is especially useful when applications need to exchange large volumes of data between computing nodes.

    5. RoCEv1 vs RoCEv2

    RoCE has two major versions: RoCEv1 and RoCEv2. Both provide RDMA communication over Ethernet, but they differ in how packets are transported across the network.

    FeatureRoCEv1RoCEv2
    TransportEthernet Layer 2UDP/IP
    Network scopeSame Layer 2 Ethernet domainCan operate across routed IP networks
    IP routingNot supported nativelySupported
    UDP destination portNot applicable4791
    Deployment flexibilityMore limitedGreater flexibility for multi-subnet environments
    Common applicationsLayer 2 RDMA networksModern data center and AI networks

    5.1 RoCEv1

    RoCEv1 operates directly over Ethernet Layer 2. It uses an Ethernet-specific transport and requires communicating endpoints to be within the same Layer 2 domain.

    This makes RoCEv1 suitable for environments where the RDMA network is designed around a single Layer 2 Ethernet domain. However, its limited routing capability can make large or geographically distributed network deployments more difficult.

    5.2 RoCEv2

    RoCEv2 encapsulates RDMA transport packets over UDP/IP. This allows RoCEv2 traffic to use IP routing and operate across different Layer 3 network segments.

    RoCEv2 uses UDP destination port 4791 and is commonly used in modern Ethernet-based data centers and AI infrastructure.

    Its use of IP routing provides greater network design flexibility, although high-performance RoCEv2 deployments still require careful consideration of routing, congestion control, packet loss, and traffic distribution.

    6. RDMA vs Traditional TCP/IP Communication

    The most important distinction between RoCE and traditional Ethernet communication is the role of RDMA.

    In a conventional TCP/IP environment, data normally passes through the application, operating system networking stack, protocol processing, and network adapter. The receiving system also processes incoming data through its networking stack.

    With RDMA, supported operations are handled by RDMA-capable network adapters, allowing data to be transferred directly between registered memory regions with reduced host CPU involvement.

    Communication StageTraditional TCP/IPRoCE with RDMA
    Application interfaceSocket APIRDMA verbs or supported RDMA frameworks
    Memory handlingConventional application and kernel memory processingRegistered memory regions
    Transport processingTCP or UDP through the networking stackRDMA transport handled by the RDMA-capable adapter
    CPU involvementRequired for conventional networking processingReduced for supported RDMA operations
    Data movementConventional networking data pathDirect memory transfer through RDMA hardware
    Typical useGeneral application communicationHigh-performance data exchange

    RDMA does not eliminate all CPU activity. Applications still need to establish connections, register memory, manage queues, and process completion events. The advantage is that the data-transfer path can avoid much of the conventional host networking overhead.

    7. Latency and CPU Utilization

    Latency and CPU utilization are two major reasons why organizations deploy RoCE in high-performance networking environments.

    7.1 Latency

    Conventional TCP/IP communication includes protocol processing, connection management, acknowledgments, and other operations that can contribute to end-to-end latency.

    RoCE reduces communication overhead by allowing RDMA-capable network adapters to handle supported data-transfer operations. This can provide lower latency and more predictable performance in appropriately configured environments.

    However, actual latency depends on network adapter capabilities, switch forwarding performance, network topology, congestion, packet size, and application behavior.

    7.2 CPU Utilization

    Traditional TCP/IP networking requires CPU resources for tasks such as protocol processing, packet handling, and memory management.

    RoCE offloads supported RDMA operations to the network adapter, reducing the CPU workload associated with data movement.

    This is particularly valuable in GPU servers, where CPU resources are also required for application coordination, storage operations, and system management.

    8. Congestion Control in RoCE Networks

    Congestion management is one of the most important considerations when deploying RoCE over Ethernet.

    Traditional TCP uses congestion control mechanisms that respond to packet loss and other network feedback. RoCEv2 does not use TCP for its RDMA transport, so a RoCE deployment requires suitable congestion-management mechanisms to maintain reliable and predictable communication.

    Common technologies include:

    • ECN (Explicit Congestion Notification): Network switches can mark packets when congestion occurs, allowing endpoints to adjust their transmission rates.

    • CNP (Congestion Notification Packet): In RoCEv2 congestion-control implementations, a receiver can send a notification to the sender when congestion is indicated.

    • PFC (Priority Flow Control): Provides priority-based pause mechanisms to help prevent packet loss for selected traffic classes. It requires careful configuration to avoid congestion spreading through the network.

    • Traffic Prioritization: Helps separate and manage different traffic classes according to application requirements.

    • Adaptive Routing: Distributes traffic across available network paths where supported.

    Not every RoCE deployment uses exactly the same congestion-control configuration. The appropriate design depends on network equipment, NIC capabilities, software support, and workload requirements.

    9. Network Requirements for RoCE

    RoCE can operate over Ethernet infrastructure, but it requires compatible hardware and appropriate network configuration.

    Important deployment requirements include:

    • RDMA-capable NICs: Network adapters must support the required RoCE version and RDMA operations.

    • Compatible Ethernet switches: Switches should support the required bandwidth, traffic handling, and congestion-management features.

    • Appropriate network configuration: VLANs, QoS, ECN, and PFC settings should be planned according to the deployment design.

    • Low-congestion topology: Network paths should provide sufficient capacity for expected traffic patterns.

    • Compatible drivers and firmware: NIC drivers, firmware, and RDMA software must work together.

    • High-speed connectivity: Optical transceivers, DAC, and AOC cables should match the port speed and transmission distance.

    Simply enabling RoCE on an Ethernet NIC does not automatically create a high-performance RDMA network. The complete system must be configured and validated.

    10. RoCE vs Traditional Ethernet: Bandwidth and Interface Speeds

    RoCE does not define a separate Ethernet port speed. It operates over supported Ethernet interfaces, meaning the available bandwidth depends on the underlying network adapter, switch, and physical link.

    Common Ethernet speeds used in RoCE deployments include:

    Ethernet SpeedPotential ApplicationNetwork Consideration
    25GSmaller server and storage deploymentsLower-bandwidth RDMA connectivity
    100GData center and storage networksHigh-speed server communication
    200GHigh-performance computingHigher-bandwidth RDMA connections
    400GAI clusters and modern data centersHigh-capacity RoCE fabrics
    800GLarge-scale AI infrastructureHigh-density next-generation connectivity
    1.6TEmerging data center infrastructureFuture high-bandwidth networking

    The choice of interface speed should be based on server capabilities, network oversubscription, expected traffic volume, and the required performance of the application.

    11. Optical Transceivers for RoCE Networks

    Optical transceivers are important in high-speed RoCE networks, especially when connecting servers to leaf switches or linking leaf and spine switches across a data center.

    Common optical connectivity options include:

    • 100G QSFP28 optical transceivers

    • 200G QSFP56 optical transceivers

    • 400G QSFP-DD optical transceivers

    • 400G QSFP112 optical transceivers

    • 800G QSFP-DD optical transceivers

    • 800G OSFP optical transceivers

    For 400G RoCE infrastructure, C-LIGHT 400G QSFP-DD optical transceivers provide multiple optical configurations for data center connectivity. Available options include SR8, DR4, FR4, and LR4 variants, with different fiber types and transmission distances.

    For higher-bandwidth Ethernet infrastructure, C-LIGHT 800G QSFP-DD optical transceivers support next-generation data center and AI networking applications. The product family includes 100m, 500m, and 2km transmission options, depending on the specific model.

    When selecting optical modules for RoCE networks, the physical form factor alone is not sufficient. The Ethernet protocol, host interface, switch compatibility, optical reach, fiber type, and supported data rate must all be verified.

    12. DAC and AOC Cables for RoCE Connectivity

    In addition to pluggable optical transceivers, DAC and AOC cables are widely used for short-distance Ethernet interconnections.

    12.1 DAC Cables

    Direct Attach Copper (DAC) cables use integrated electrical connections to link compatible network ports. They are commonly used for short-distance connections between servers and switches, particularly within the same rack.

    DAC cables can provide a practical option where transmission distance is limited and the equipment supports the required electrical interface.

    12.2 AOC Cables

    Active Optical Cables (AOCs) integrate optical transmitters and receivers into a cable assembly. They provide optical transmission over short distances without requiring separate pluggable optical transceivers and fiber patch cables.

    For 400G RoCE applications, C-LIGHT 400G AOC cables include configurations identified for 400G InfiniBand and Ethernet-related connectivity. The exact product configuration should be selected according to the target protocol and equipment compatibility.

    For next-generation 800G environments, C-LIGHT 800G AOC cables include both InfiniBand and Ethernet configurations. The product family supports short-reach connections using QSFP-DD and OSFP interfaces, with different breakout options.

    It is important to distinguish Ethernet and InfiniBand configurations when selecting an AOC. A similar connector or form factor does not guarantee protocol compatibility.

    13. RoCE vs Traditional Ethernet: Application Scenarios

    ApplicationRoCETraditional Ethernet
    AI model trainingHighly suitable for RDMA-based communicationSuitable depending on workload and performance requirements
    GPU cluster communicationSuitable for high-speed data exchangeCommonly used for general server communication
    High-performance storageSuitable for supported RDMA storage protocolsSuitable for conventional TCP/IP storage access
    Database applicationsUseful for selected high-performance database workloadsWidely used
    Enterprise applicationsUsually unnecessary for general workloadsWidely used
    Cloud computingSuitable for specialized high-performance infrastructureCommon general-purpose networking foundation
    Scientific computingSuitable for RDMA-intensive applicationsSuitable for general communication requirements
    VirtualizationPossible with compatible platforms and configurationsCommon

    RoCE is most valuable when applications can take advantage of RDMA. For general-purpose enterprise communication, traditional Ethernet is often simpler and sufficient.

    14. When Should You Use RoCE Instead of Traditional Ethernet?

    The decision depends on application requirements rather than network speed alone.

    Choose RoCE When:

    • Your application requires low-latency communication between servers.

    • Large volumes of data must be transferred with reduced CPU overhead.

    • You are building an AI training or GPU computing cluster.

    • Your storage platform supports RDMA-based data access.

    • You have RDMA-capable NICs and compatible Ethernet switches.

    • Your network team can configure and monitor the required congestion-management mechanisms.

    Choose Traditional Ethernet When:

    • You are deploying general enterprise applications.

    • Your workloads primarily use conventional TCP/IP communication.

    • Application compatibility and operational simplicity are priorities.

    • RDMA capabilities are not required.

    • Your existing Ethernet infrastructure already meets application performance requirements.

    For some data centers, both technologies can coexist. General application traffic can use conventional TCP/IP Ethernet, while selected high-performance workloads use RoCE over dedicated or logically separated Ethernet resources.

    15. C-LIGHT Optical Connectivity for RoCE Networks

    As AI and high-performance computing workloads increase network bandwidth requirements, optical connectivity becomes an important part of RoCE infrastructure.

    C-LIGHT provides optical transceivers and high-speed interconnect solutions for data centers, AI networks, and high-performance computing environments.

    When building a RoCE network, selecting the correct optical connectivity requires checking the Ethernet interface speed, NIC and switch compatibility, transmission distance, fiber type, and cable configuration.

    16. Frequently Asked Questions

    Q1: What is the main difference between RoCE and Traditional Ethernet?

    RoCE enables RDMA communication over Ethernet, reducing CPU involvement in supported data transfers. Traditional Ethernet commonly uses TCP/IP communication through the operating system networking stack.

    Q2: Is RoCE a separate Ethernet standard?

    No. RoCE is a technology that enables RDMA over Ethernet networks. It does not replace Ethernet as the underlying network technology.

    Q3: Does RoCE use TCP?

    No. RoCEv1 operates directly over Ethernet Layer 2, while RoCEv2 uses UDP/IP. Neither uses TCP for its RDMA transport.

    Q4: What is the difference between RoCEv1 and RoCEv2?

    RoCEv1 operates at Ethernet Layer 2, while RoCEv2 encapsulates RDMA traffic over UDP/IP and supports IP routing.

    Q5: Does RoCE reduce CPU utilization?

    Yes. RDMA-capable network adapters handle supported memory-transfer operations in hardware, reducing the CPU processing required for data movement.

    Q6: Does RoCE require PFC?

    PFC is used in some RoCE deployments, but it is not a universal requirement. The congestion-management design depends on the network equipment and deployment architecture.

    Q7: Can RoCE work with standard Ethernet switches?

    RoCE can operate over Ethernet switches, but the switches and network configuration must support the required performance and traffic-management features.

    Q8: Is RoCE suitable for AI data centers?

    Yes. RoCE is widely used in Ethernet-based AI infrastructure because it supports efficient RDMA communication between GPU servers and computing nodes.

    Q9: What optical transceivers are used in RoCE networks?

    Depending on network bandwidth and transmission distance, RoCE deployments may use 100G, 200G, 400G, and 800G Ethernet-compatible optical transceivers.

    Q10: Can RoCE and Traditional Ethernet operate in the same data center?

    Yes. A data center can use conventional TCP/IP Ethernet for general applications and RoCE for selected high-performance workloads, provided the network is properly designed and configured.

    17. Conclusion

    RoCE and traditional Ethernet share the same Ethernet foundation, but they use different communication mechanisms. Traditional Ethernet commonly relies on TCP/IP to provide general-purpose network connectivity, while RoCE enables RDMA operations over Ethernet to reduce CPU overhead and improve data-transfer efficiency.

    RoCE is particularly valuable for AI training, GPU clusters, high-performance storage, and other workloads that require low-latency communication and efficient data movement. However, it requires RDMA-capable network adapters, compatible switches, and carefully designed congestion management.

    As data center networks continue to evolve toward 400G, 800G, and 1.6T connectivity, high-speed optical transceivers, DAC cables, and AOCs will remain important components of modern Ethernet infrastructure. Choosing the right connectivity solution helps ensure that the physical network can support the performance requirements of both conventional Ethernet and RDMA-based RoCE applications.

    For any questions, please contact us by email or WhatsApp.

    Email: sales@c-light.com

    WhatsApp: +86 132 6656 7067

    Related Articles

    Call
    Top