C-LIGHT telephone TEL:+86 132 6656 7067    
Language
C-LIGHT search

RoCEv2 vs TCP/IP:Differences in RDMA, Latency and Network Performance

By C-LIGHT Marketing 丨 Jul 17, 2025
Table of Contents

    As data center workloads become more demanding, network performance is increasingly important for AI training, distributed computing, high-performance storage, and large-scale cloud infrastructure. Traditional TCP/IP communication remains the foundation of most network applications, while RoCEv2 (RDMA over Converged Ethernet version 2) provides an alternative data transfer mechanism designed to reduce communication overhead and latency.

    Although both technologies operate over IP-based Ethernet networks, they serve different purposes. TCP provides reliable, connection-oriented data delivery, whereas RoCEv2 enables Remote Direct Memory Access (RDMA) over routable Ethernet. Understanding their differences helps network architects select the right communication method for different workloads.

    1. What Is RoCEv2?

    RoCEv2 stands for RDMA over Converged Ethernet version 2. It allows servers to exchange data directly between application memory regions over an Ethernet network, reducing the need for conventional operating system networking operations during data transfer.

    Unlike RoCEv1, which operates at Layer 2 and is limited to a single Ethernet broadcast domain, RoCEv2 encapsulates RDMA transport packets in UDP/IP. This makes RoCEv2 routable across Layer 3 networks and more suitable for larger data center fabrics.

    RoCEv2 uses UDP destination port 4791. The UDP header provides encapsulation for RDMA traffic; it does not mean that RoCEv2 behaves like an ordinary application using best-effort UDP communication. RDMA transport services can provide their own delivery and reliability mechanisms.

    1.1 How RoCEv2 Works

    RoCEv2 relies on RDMA-capable network adapters, commonly called RNICs. These adapters handle much of the RDMA communication in hardware, allowing data to move between memory regions on different servers with reduced CPU involvement.

    A simplified RoCEv2 communication process is:

    1. The application registers or prepares memory regions for RDMA operations.

    2. The sender posts an RDMA work request to its RNIC.

    3. The RNIC processes the operation and encapsulates the RDMA transport packet in UDP/IP and Ethernet headers.

    4. Ethernet switches and IP routers forward the packet through the network.

    5. The destination RNIC processes the RDMA operation and places the data into the appropriate memory region.

    6. The receiving application can access the transferred data according to the configured RDMA operation and completion mechanism.

    Because much of the data movement is handled by the RNIC, RoCEv2 can reduce CPU overhead and communication latency in workloads that benefit from RDMA.

    2. What Is TCP/IP?

    TCP/IP is the fundamental protocol suite used by the Internet and most enterprise networks. IP handles packet addressing and forwarding, while TCP provides reliable, ordered, connection-oriented data delivery between applications.

    In a conventional TCP/IP application, data is passed through the operating system's networking stack. TCP divides the data stream into segments, tracks sequence numbers, acknowledges received data, and retransmits lost segments when necessary.

    TCP is widely used because it provides a mature and interoperable communication model. Applications such as web services, databases, file transfers, enterprise software, and cloud platforms commonly rely on TCP-based communication.

    2.1 How TCP/IP Communication Works

    A typical TCP communication process includes:

    1. The sender establishes a TCP connection with the destination.

    2. The application passes data to the TCP stack.

    3. TCP segments the data and adds sequence and control information.

    4. IP delivers the packets across the network.

    5. The receiver acknowledges received data and reassembles the ordered byte stream.

    6. If data is lost, TCP detects the loss and retransmits the missing information.

    This process provides reliable data delivery across a wide range of network conditions. However, conventional TCP communication may involve additional memory copies, protocol processing, interrupts, and CPU scheduling, depending on the operating system, hardware, and application implementation.

    3. RoCEv2 vs TCP/IP: Main Differences

    The most important distinction is that RoCEv2 provides RDMA-based communication, while conventional TCP/IP applications generally exchange data through the TCP socket interface. The comparison below focuses on RoCEv2 RDMA communication versus traditional TCP-based communication.

    Comparison PointRoCEv2Traditional TCP/IP
    Primary purposeLow-overhead RDMA data transferGeneral-purpose network communication
    Transport encapsulationRDMA transport over UDP/IPTCP segments over IP
    Data accessSupports direct memory access through RDMA operationsUses conventional application and operating system networking interfaces
    CPU involvementCan be significantly reduced through RNIC hardware offloadUsually requires more host networking stack processing, depending on offload capabilities
    Data orderingDepends on the selected RDMA transport serviceTCP provides ordered byte-stream delivery
    ReliabilityReliable RDMA transport modes provide their own acknowledgment and retransmission mechanismsTCP uses sequence numbers, acknowledgments and retransmissions
    Congestion managementMay use ECN-based mechanisms such as DCQCN, with PFC in some deploymentsUses TCP congestion control algorithms and feedback from network conditions
    Network requirementsRequires compatible RNICs, switches and carefully configured network behaviorSupported by a broad range of standard IP networking equipment
    Typical applicationsAI clusters, HPC, distributed storage and low-latency data transferWeb services, databases, file transfers and general enterprise applications
    Deployment complexityHigher, especially in large-scale RDMA fabricsGenerally lower and more straightforward

    4. RDMA vs TCP: How Data Transfer Differs

    The most significant architectural difference between RoCEv2 and conventional TCP communication is how data moves between applications and network adapters.

    4.1 Direct Memory Access with RoCEv2

    RDMA allows a network adapter to transfer data directly between registered memory regions on two systems. The RNIC handles key transport operations without requiring the CPU to process every data movement step through the traditional networking stack.

    This reduces software processing overhead and can help applications achieve lower latency and higher throughput, particularly when transferring large amounts of data between servers.

    4.2 Conventional Data Transfer with TCP/IP

    Traditional TCP applications use sockets to send and receive data. The operating system manages TCP connections, packet processing, acknowledgments, retransmissions, and delivery of the ordered byte stream to the application.

    Modern systems can reduce this overhead through technologies such as TCP Segmentation Offload (TSO), Generic Receive Offload (GRO), Receive Side Scaling (RSS), kernel bypass, and other hardware acceleration features. Therefore, TCP performance should not be evaluated based on an assumption that every packet requires the same amount of CPU processing.

    5. Latency and CPU Utilization

    Latency and CPU utilization are two major reasons organizations consider RoCEv2 for demanding data center workloads.

    5.1 Latency

    RoCEv2 can achieve very low communication latency by reducing software processing and enabling hardware-assisted data transfer. This is especially useful when applications exchange data frequently and are sensitive to communication delays.

    TCP also supports high-performance communication, but its conventional connection and reliability mechanisms introduce processing steps that may increase latency in some workloads. Actual performance depends on the network adapter, operating system, application design, congestion level, packet size, and network topology.

    5.2 CPU Utilization

    With RoCEv2, RNIC hardware handles much of the RDMA data movement and transport processing. This can free CPU resources for application workloads, which is valuable in GPU servers where CPU resources are also needed for orchestration, data preparation, and system management.

    TCP can also benefit from NIC offloads and optimized networking stacks. However, conventional TCP socket communication generally involves more host software processing than a well-configured RDMA data path.

    6. Reliability and Packet Loss Handling

    Both RoCEv2 and TCP can support reliable data delivery, but their reliability mechanisms are different.

    6.1 TCP Reliability

    TCP provides a reliable, ordered byte stream. It uses sequence numbers and acknowledgments to track received data. When packet loss is detected, TCP can retransmit missing segments and adjust its sending behavior according to congestion signals.

    6.2 RoCEv2 Reliability

    RoCEv2 uses RDMA transport services rather than TCP. For example, Reliable Connected (RC) transport provides ordered delivery and transport-level reliability mechanisms, including acknowledgments and retransmissions. Other RDMA transport types have different delivery guarantees.

    This distinction matters: RoCEv2 uses UDP/IP for network-layer encapsulation, but reliable RDMA transport modes do not simply inherit UDP's best-effort delivery behavior. Reliability is handled by the RDMA transport layer.

    7. Congestion Control: DCQCN, ECN and TCP Congestion Control

    Congestion management is especially important in AI clusters and high-performance data center fabrics. Multiple servers may send large volumes of traffic toward the same destination, creating temporary or sustained congestion at switch ports.

    7.1 TCP Congestion Control

    TCP uses congestion control algorithms to regulate the sender's transmission rate. Depending on the algorithm and network configuration, TCP can respond to packet loss, acknowledgments, and Explicit Congestion Notification (ECN) signals.

    This general-purpose congestion control model is widely deployed and works across many types of IP networks.

    7.2 RoCEv2 Congestion Control

    RoCEv2 deployments commonly use ECN-based congestion management. In a typical DCQCN (Data Center Quantized Congestion Notification) design, switches mark packets when congestion is detected. The receiving RNIC can generate a Congestion Notification Packet (CNP), which informs the sender that its transmission rate should be adjusted.

    Priority Flow Control (PFC) may also be configured to reduce packet loss for selected traffic priorities. However, PFC is not a universal requirement for every RoCEv2 deployment. The appropriate combination of ECN, PFC, queue management, and traffic engineering depends on the network design and vendor implementation.

    RoCEv2 performance depends heavily on consistent end-to-end configuration. Poor congestion management can lead to queue buildup, packet loss, reduced throughput, or congestion spreading across the fabric.

    8. Network Infrastructure and Deployment Requirements

    RoCEv2 and TCP/IP can both operate over Ethernet infrastructure, but their deployment requirements are not identical.

    8.1 RoCEv2 Infrastructure

    A RoCEv2 network typically requires:

    • RDMA-capable network adapters with appropriate RoCEv2 support.

    • Ethernet switches that support the required traffic classification and congestion-management features.

    • Correct configuration of IP addressing, routing, MTU, QoS and traffic priorities.

    • Consistent ECN and congestion-control settings across the network where required.

    • Validation of RNIC, switch firmware, driver and operating system compatibility.

    RoCEv2 is routable, but routability does not automatically guarantee optimal performance. Network design and configuration remain important, especially when traffic crosses multiple switch layers.

    8.2 TCP/IP Infrastructure

    TCP/IP works across a much broader range of standard networking environments. Most Ethernet switches, routers, servers, operating systems, and application frameworks support TCP/IP without requiring specialized RDMA hardware.

    This makes TCP/IP easier to deploy, troubleshoot, and integrate with existing enterprise applications.

    9. RoCEv2 vs TCP/IP in AI Data Centers

    AI training and large-scale GPU computing generate substantial communication traffic between servers. Distributed training frameworks frequently exchange model parameters, gradients, and intermediate data across multiple GPUs and nodes.

    For these workloads, communication overhead can become a major performance factor. RoCEv2 is commonly considered for Ethernet-based GPU clusters because RDMA can reduce CPU involvement and provide low-latency communication between compatible systems.

    TCP/IP remains important for application services, management traffic, storage access, orchestration, and other general-purpose communications. In many AI data centers, RoCEv2 and TCP/IP coexist rather than replace one another.

    9.1 When RoCEv2 Is a Good Fit

    • Distributed AI training with frequent server-to-server data exchange.

    • GPU clusters requiring low-latency communication.

    • High-performance computing applications.

    • RDMA-based distributed storage and data services.

    • Workloads where CPU overhead from conventional networking is a limiting factor.

    9.2 When TCP/IP Is a Good Fit

    • Web applications and application servers.

    • Database connections and enterprise software.

    • General file transfers and network services.

    • Cloud management, orchestration and control-plane communication.

    • Environments where broad compatibility and operational simplicity are priorities.

    10. Optical Connectivity for RoCEv2 and High-Speed Ethernet

    RoCEv2 uses Ethernet and IP networking, so its physical connectivity can rely on standard Ethernet-compatible optical transceivers, DAC cables, and AOCs. The selected optical interface must match the Ethernet speed, reach, fiber type, connector, and switch or server port requirements.

    It is important to distinguish the physical layer from the transport protocol. An optical transceiver carries Ethernet signals over an optical link; it does not itself implement RoCEv2, RDMA, or TCP. Protocol support is determined by the network adapters, switches, drivers, and software stack.

    10.1 400G and 800G Optical Transceivers

    As AI clusters and data center fabrics move toward higher Ethernet speeds, 400G and 800G optical transceivers provide the bandwidth needed for high-density switch-to-switch and server-to-network connections.

    C-LIGHT offers 400G QSFP-DD optical transceivers for high-speed Ethernet connectivity. For newer high-bandwidth deployments, the 800G QSFP-DD optical transceiver portfolio supports data center and AI networking applications, with configurations covering different transmission distances.

    10.2 Active Optical Cables for Short-Reach Connections

    For short-distance connections between switches and servers, Active Optical Cables (AOCs) provide an integrated optical interconnect solution. They can simplify cabling in high-density environments where copper cable reach, routing, or cable management becomes a concern.

    C-LIGHT's 800G AOC solutions include Ethernet and InfiniBand-oriented configurations. When selecting a cable, the exact interface type and protocol configuration should be checked against the equipment being connected.

    11. How to Choose Between RoCEv2 and TCP/IP

    The right choice depends on application requirements rather than bandwidth alone. A high-speed Ethernet link does not automatically require RoCEv2, and TCP/IP can remain the better option for many applications.

    RequirementRecommended ApproachReason
    Low-latency GPU communicationRoCEv2RDMA can reduce software overhead and support efficient data movement.
    Distributed AI trainingRoCEv2 or optimized alternativesRDMA-based communication can benefit workloads with intensive node-to-node data exchange.
    General web applicationsTCP/IPBroad application support and mature reliability mechanisms.
    Enterprise database accessUsually TCP/IPMost database clients and services are designed around conventional network communication.
    RDMA-based storageRoCEv2 where supportedCan reduce data movement overhead for compatible storage and server platforms.
    Mixed enterprise network trafficTCP/IP, with RoCEv2 for selected workloadsDifferent traffic types can use the same Ethernet infrastructure with appropriate design and configuration.

    Before deploying RoCEv2, organizations should verify RNIC support, switch capabilities, congestion-management behavior, routing design, and application compatibility. For many environments, a hybrid architecture is more practical than attempting to move all network traffic to RDMA.

    12. Conclusion

    RoCEv2 and TCP/IP address different communication requirements. RoCEv2 extends RDMA capabilities over routable Ethernet, enabling low-overhead data transfer between compatible systems. TCP/IP provides a general-purpose communication foundation with broad compatibility, mature reliability, and straightforward deployment.

    For AI clusters, HPC environments, and RDMA-based storage, RoCEv2 can improve communication efficiency when the hardware and network are properly configured. For web services, databases, enterprise applications, and general network traffic, TCP/IP remains a practical and widely supported choice.

    In modern data centers, the two technologies are often complementary. Selecting the right protocol for each workload—and matching it with suitable Ethernet switches, network adapters, and optical connectivity—is more effective than treating RoCEv2 and TCP/IP as competing replacements.

    Frequently Asked Questions

    Q1: Is RoCEv2 the same as TCP/IP?

    No. RoCEv2 is an RDMA transport technology encapsulated in UDP/IP, while TCP is a reliable transport protocol within the TCP/IP suite.

    Q2: Does RoCEv2 use TCP?

    No. RoCEv2 uses UDP/IP encapsulation rather than TCP. Its RDMA transport layer provides the relevant transport services.

    Q3: Does RoCEv2 use UDP?

    Yes. RoCEv2 encapsulates RDMA transport packets in UDP/IP and uses UDP destination port 4791.

    Q4: Is RoCEv2 more reliable than TCP?

    Not inherently. Reliable Connected RDMA provides its own reliability mechanisms, while TCP provides reliable, ordered byte-stream delivery. Their reliability models are different.

    Q5: Why is RoCEv2 used in AI data centers?

    RoCEv2 can reduce CPU involvement and communication overhead, making it useful for distributed GPU workloads that exchange large amounts of data.

    Q6: Does RoCEv2 require a lossless Ethernet network?

    RoCEv2 deployments often use congestion-management features to limit packet loss. PFC is common in some designs but is not a universal requirement for every implementation.

    Q7: What is the difference between RoCEv1 and RoCEv2?

    RoCEv1 operates directly over Ethernet Layer 2, while RoCEv2 uses UDP/IP encapsulation and can be routed across Layer 3 networks.

    Q8: Can RoCEv2 and TCP/IP run on the same Ethernet network?

    Yes. They can coexist on shared Ethernet infrastructure when traffic classification, QoS, congestion management, and bandwidth allocation are properly designed.

    Q9: Does RoCEv2 require special optical transceivers?

    Not specifically. It uses Ethernet-compatible physical links, but the optical transceiver must match the required Ethernet speed, reach, fiber type, and equipment interface.

    Q10: Can TCP/IP replace RoCEv2 in GPU clusters?

    TCP-based communication can support GPU clusters, but it may introduce different CPU and latency overheads. The best option depends on workload characteristics, hardware support, and performance requirements.

    For any questions, please contact us by email or WhatsApp.

    Email: sales@c-light.com

    WhatsApp: +86 132 6656 7067

    Related Articles

    Call
    Top