GPU Networks Read via RDMA and NCCL: InfiniBand, RoCE, Kubernetes Deployment

Status: Draft (Updated 2026-10-06). Based on descriptions verified with official documentation, and only the parts directly observed with software RoCE (rxe) (section "Field Measurement" at the end of Section 10) are values measured in a disposable virtual machine. There are no measurements using actual RDMA hardware and GPUs. Please read "Verified Versions and Dates" and "Unverified Items" at the end of this article first. Writing Rules: Field measurement data is the default, and items that could not be measured are marked as "Unmeasured".

Table of Contents

• Summary
• 1. Why TCP Is Not Enough
• 2. InfiniBand and RoCE v2
• 3. Layers and Bandwidth
• 4. GPUDirect RDMA and GPUDirect Storage
• 5. NCCL
• 6. How to Measure
• 7. Common Failures
• 8. RDMA in Kubernetes
• 9. Where RDMA Is Needed in Inference (Connected to Part 6)
• 10. Try It Yourself
  • Field Measurement: Reproducing verbs flow and two types of failures using software RoCE (rxe)
• External Evidence: Verified with papers and official materials
• 11. Interview Questions
• 12. Common Misconceptions
• References (Actually Opened)
• Verified Versions and Dates, and Unverified Items

Summary

When multiple GPUs compute a single model together, communication equals performance. Inside the server, NVLink handles this, while outside the server, InfiniBand or RoCE (RDMA over Converged Ethernet) networks supporting RDMA (Remote Direct Memory Access) take charge. NCCL (NVIDIA Collective Communications Library) detects this layer and orchestrates collective communications (such as all-reduce). This post covers layer-by-layer bandwidth differences, why RoCE is difficult, NCCL environment variables and measurement tools, troubleshooting sequences, and Kubernetes configurations.

What You Can Explain in an Interview After Reading This Post

1. Why TCP is insufficient and the differences between InfiniBand and RoCE v2 (specifically, what PFC and ECN do in RoCE and symptoms when configurations are incorrect).
2. How NCCL performs collective communication, nccl-tests how to read its busbw, and the troubleshooting sequence when multi-node performance drops.
3. The components required for pods to use RDMA in Kubernetes, and the points where RDMA is needed in inference (KV transfer, expert parallelism, model loading).

Author's Background. I am experimenting in a standard Ethernet (1-2.5GbE class) environment and have no experience with RDMA, InfiniBand, NVLink, or VAST. This post is compiled from reading official documentation and is not a record of operational experience. The hands-on section is limited to what is feasible, and unexecuted parts are marked as such.

1. Why TCP Is Not Enough

According to the Dynamo documentation, RDMA allows network adapters to move data from one host's memory to another host's memory (or GPU-to-GPU when using GPUDirect RDMA) without going through the CPU, kernel, or TCP/IP stack. TCP typically involves copying between user buffers and kernel buffers, system calls, and interrupts. RDMA enables kernel bypass and zero-copy because the NIC directly reads and writes to memory whose pages are pinned in advance through pre-registration (ibv_reg_mr). The speed difference is also mathematically massive. 1GbE is 0.125GB/s, and a 400Gb/s NIC is 50GB/s, which is 400 times faster. Therefore, TCP serves as a fallback path when RDMA is absent (NCCL_IB_DISABLE=1 NCCL falls back to IP sockets if this occurs).

2. InfiniBand and RoCE v2

RoCE v1 uses only the Ethernet link layer for a single broadcast domain, whereas v2 adds UDP/IP to enable routing (Wikipedia).

Why RoCE Is Difficult. Because RDMA is vulnerable to packet loss, loss must be prevented or mitigated in Ethernet. NVIDIA Cumulus documentation explains two methods. PFC (Priority Flow Control) applies hop-by-hop pauses per priority to prevent buffer overflows, while ECN (Explicit Congestion Notification) marks congested packets at the switch so the receiving NIC sends back a CNP (Congestion Notification Packet) to lower the transmission rate. By default, RoCE data uses priority 3 and CNP uses strict priority queue 6. Onyx documentation distinguishes between lossless (PFC across the entire network), semi-lossless (PFC only between hosts and top-of-rack switches), and lossy (no PFC) modes. DCQCN (SIGCOMM 2015) is an ECN-based RoCE v2 congestion control designed to reduce the side effects of PFC; according to the abstract, it is implemented in Mellanox NICs and deployed in Microsoft datacenters. DCQCN does not replace PFC. Section 7 of the paper notes that when the loss rate exceeds 0.1%, throughput plummets, making PFC still necessary. PFC can cause network-wide deadlocks (a 2016 Microsoft paper reports an actual deadlock caused by a combination of PFC and packet flooding for unknown MAC addresses in a test cluster, along with an incident where a single NIC continuously sent PAUSE frames, blocking the entire network; see "External Evidence" below), and it is noted that Ethernet DCB configurations are more complex than InfiniBand (Wikipedia).

Failures appear silently. The MinIO MemKV runbook (third-party material, not NVIDIA documentation) notes that PFC priority, DSCP mapping, and MTU must match, and if the Ethernet MTU is 1500, the active RoCE MTU is negotiated down to 1024, and if misconfigured, it ends with IBV_WC_RETRY_EXC_ERR , which only becomes apparent between nodes.

3. Layers and Bandwidth

 노드(예: 8 GPU)
 GPU0 ... GPU7 ══ NVLink / NVSwitch ══ (노드 안 전체 연결)
   │PCIe          │PCIe
  NIC0    ...    NIC7 ──> 같은 번호 NIC 끼리 같은 리프 스위치 (rail)

Converting NVLink 4th Generation per direction (half) yields about 450GB/s, which is roughly 9 times that of a single NIC (author's conversion). The H800 cluster in the DeepSeek-V3 report also had NVLink at 160GB/s versus InfiniBand at 50GB/s, and this gap drives the trend from Part 6 where "TP is inside the node, while PP and EP are between nodes." Systems like NVL72 change the boundary as 72 GPUs form a single NVLink domain. The rail structurewas confirmed in the DGX SuperPOD reference architecture, where identically numbered NICs in each server are connected to the same leaf switch so that servers on the same rail traverse only a single switch.

4. GPUDirect RDMA and GPUDirect Storage

GPUDirect RDMA allows NICs to directly read and write GPU memory, eliminating the host memory hop. Official documentation prerequisites state that it is optimal when two devices share the same PCIe root complex with only PCIe switches in between (performance degrades if crossing QPI/UPI), and IOMMU must be disabled or set to 1:1 passthrough. NCCL documentation guides users to load the nvidia-peermem module on every boot or use the open-source driver's DMA-BUF path (unloaded modules and ACS are also common causes), and restricts the maximum distance (LOC, PIX, PXB, PHB, SYS) between the NIC and GPU using NCCL_NET_GDR_LEVEL .

GPUDirect Storage (GDS) eliminates CPU bounce buffers via DMA between storage and GPU memory, supporting cuFile APIs along with local NVMe, NVMe-oF, Lustre, WekaFS, and NFS over RDMA. If conditions are not met, it falls back to a compatibility mode that passes through CPU memory, destroying the performance benefit.

5. NCCL

Operations. Collective operations in the NCCL 2.32.3 documentation include AllReduce, Broadcast, Reduce, AllGather, ReduceScatter, AlltoAll, Gather, and Scatter, along with point-to-point send/recv. All ranks must call them with the same count and datatype; mismatches lead to undefined behavior such as hangs, crashes, or data corruption.

Algorithms and Topologies. NCCL_ALGO Algorithms listed in the documentation are Ring, Tree, CollnetChain, CollnetDirect, NVLS, NVLSTree, and PAT (NVLS is NVLink SHARP offload). According to the NCCL 2.4 blog, the ring utilizes full bandwidth but latency is proportional to the number of GPUs, whereas the double binary tree maintains bandwidth while latency increases logarithmically, improving latency by up to 180 times compared to the ring on Summit with 24,576 GPUs (NVIDIA's claimed measurement). NCCL detects NVLink, PCIe, and NIC connections during initialization to build a graph, and outputs the detection results via NCCL_TOPO_DUMP_FILE .

The documentation warns against leaving debugging and tuning variables in production settings, noting that a gain in one benchmark may be a loss elsewhere. In inference, NCCL is used for layer-wise all-reduce in TP, and EP routes differ depending on the backend (allgather_reducescatter, DeepEP, etc.). KV transfer is handled by NIXL/UCX, so NCCL variables do not apply (Part 6).

6. How to Measure

./build/all_reduce_perf -b 8 -e 128M -f 2 -g 8                    # 노드 안 GPU 8장
mpirun -np 64 -N 8 ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1  # 8노드 x 8 GPU (MPI=1 빌드)
ib_write_bw -d mlx5_0 -a                       # perftest 서버
ib_write_bw -d mlx5_0 <서버주소> -a            # 클라이언트. --use_cuda=0 을 더하면 GPU 메모리 경로
ib_write_lat -d mlx5_0 <서버주소>              # 꼬리 지연 확인

nccl-tests outputs time, algbw(크기/시간), and busbw. busbw is a value multiplied by an operation-specific correction factor to compare against hardware limits regardless of the number of ranks; for AllReduce, it is algbw × 2(n−1)/n, and for ReduceScatter, AllGather, and AlltoAll, it is algbw × (n−1)/n . You look at how close the busbw of large messages is to the NVLink or NIC theoretical limits, and for small messages, you look at time (latency). The order in the NCCL documentation is to look at intra-node nvbandwidth and nvidia-smi topo -m, and inter-node ib_write_bw/ib_write_lat first, and if these values match expectations but NCCL alone is slow, then suspect the NCCL configuration.

7. Common Failures

8. RDMA in Kubernetes

flowchart TB
  subgraph Node[GPU 노드]
    OFED[MOFED 드라이버<br/>ofedDriver]
    PEER[nvidia_peermem<br/>GPU Operator]
    DP[RDMA 공유 디바이스 플러그인]
    MUL[Multus + NetworkAttachmentDefinition]
  end
  OFED --> DP
  DP -->|rdma/hca_shared_devices_a 광고| KL[kubelet]
  KL --> Pod[파드: GPU + rdma 자원 + 보조 인터페이스 net1]
  MUL --> Pod
  PEER --> Pod

NVIDIA Network Operator (latest v26.4.2, 2026-09-16) enables sub-configurations with a single NicClusterPolicy(README notation is NICClusterPolicy, name is nic-cluster-policy fixed as). The items include the MOFED driver container (ofedDriver), rdmaSharedDevicePlugin, sriovDevicePlugin, secondaryNetwork(Multus, containernetworking plugin, IPoIB CNI), nvIpam, nicConfigurationOperator and others, and the MacvlanNetwork, HostDeviceNetwork, IPoIBNetwork CRD is converted NetworkAttachmentDefinition into.

• RDMA shared device plugin(v1.5.4) uses resourceName and the default prefix rdma to advertise resources like rdma/hca_shared_devices_a , and rdmaHcaMax pods share a single HCA. Because it is a shared method, isolation or bandwidth guarantees are difficult to expect (author's interpretation). SR-IOV device plugin advertises network resources such as VFs (Virtual Functions), and VFs must be created in advance.
• Host network vs. auxiliary interfaces. hostNetwork uses the node's NIC directly, making it simple, but since it shares the node network, port conflicts can occur (author's note). Multus adds eth0 external interfaces (net1 , etc.) to separate the RDMA network from the default CNI.
• GPU side. Dynamo's AKS guide installs the GPU Operator via driver.rdma.enabled=true, driver.rdma.useHostMofed=true to bring up nvidia_peermem and requests rdma/hca_shared_devices_a: 1 for pods. This guide states that setting memlock to unlimited on the nodes eliminates the need for IPC_LOCK , whereas the vLLM documentation recommends adding IPC_LOCK , so choose the approach that fits your environment.
• Topology-aware placement. Dynamo guides the use of Grove and KAI Scheduler for topology-aware gang scheduling, and allows you to kvTransferPolicy(experimental) prefer decodes in the same zone or rack as prefill. The reason why a GPU and its nearby NIC must be provisioned together is due to NCCL_NET_GDR_LEVEL distance limitations and the rail-optimized architecture, though we were unable to confirm how to allocate them as a matched pair in Kubernetes.

9. Where RDMA is Needed in Inference (Connecting to Part 6)

NVIDIA's Dynamo documentation states that moving KVs over TCP results in a TTFT of about 98 seconds, and over RDMA, 200–500 ms. There are conditions mentioned within the document set. The 98-second figure is for Llama-3.1-8B, 8,000 input tokens, on an AWS p5.48xlarge when NIXL on an EFA pod silently falls back to UCX's slow path,and the 200–500 ms value is marked by the same document as a "specification-based expectation" (the actual measured additional KV transfer overhead with EFA under the same conditions was 37 ms). It should not be read as a general comparison between TCP and RDMA (Part 6 "External Evidence").

10. Hands-on Tryout

Since there is no RDMA hardware, this is in the realm of software and TCP. Items 1 and 2 were not run directly, and item 3 (software RoCE) records the results executed directly in "Actual Measurement" below.

1. iperf3 to check Ethernet limits. Run iperf3 -s and iperf3 -c <서버> -P 4 -t 10 on two servers and compare them with the 1GbE and 2.5GbE theoretical values in the table above. Avoid control plane (etcd) nodes and perform this during off-peak hours. Direct execution result: Across three different node pairs (pod network, 10 seconds), a single stream was 894–902 Mbit/s and 4 streams were 901–912 Mbit/s, both being identical (0 retransmissions), which is about 90–91% of 1GbE wire speed (about 112 MB/s). Moving KVs over this network takes about 8 seconds for 8,000 tokens (about 917 MB for a 1.7B model at 114,688 bytes per token), and about 19 seconds for about 2.1 GB for a 32B model at 256 KiB per token (calculated). Since the same 1.7B model takes about 1 second (8.2k tok/s, Part 21) to process an 8,000-token input on an 8GB-class GPU, In this network, recomputing KVs is faster than transferring them(comparison between measurements in parts 14 and 21, and this calculation).
2. TCP socket NCCL. On two servers with a single GPU each (e.g., hostA, hostB), MPI=1 built with nccl-tests directly on the hosts. Make sure no other workloads are using the GPUs first. mpirun Check first that no other workloads are using the GPUs.

   mpirun -np 2 -H hostA:1,hostB:1 -x NCCL_IB_DISABLE=1 -x NCCL_SOCKET_IFNAME=<인터페이스> \
     -x NCCL_DEBUG=INFO ./build/all_reduce_perf -b 8 -e 256M -f 2 -g 1
   

   The expectation is that socket transfers will appear in the logs and the busbw for large messages will not exceed the link speed (0.125GB/s for 1GbE), rather than actual measurements. A single node with one GPU has no communication partner, so it is meaningless.
3. Observing verbs flow with Soft-RoCE (rxe). According to RHEL documentation, Soft-RoCE is a Technology Preview and is scheduled for removal in RHEL 10. Performance is bottlenecked by the CPU so the numbers are meaningless; it is used for checking concepts like device enumeration, port states, and MTU. Since it creates kernel modules and RDMA devices, do this on a disposable VM rather than a production node.

   sudo rdma link add rxe0 type rxe netdev <이더넷 인터페이스>
   rdma link show; ibv_devices; ibv_devinfo -d rxe0 | grep -E "state|active_mtu"
   ib_write_bw -d rxe0 -a          # VM 두 대에 rxe 를 만들고 서버/클라이언트로
   sudo rdma link delete rxe0      # 정리 (man rdma-link 로 확인)
   

Measurement: Reproducing both verbs flow and failures using Soft-RoCE (rxe)

Environment. Both server and client were run within a single disposable virtual machine (4 vCPU, 4GiB, Linux 6.8, virtual NIC MTU 1500) by connecting to its own address. rdma-core, ibverbs-utils, perftest(ib_write_bw, ib_write_lat), iperf3 was installed. The default cloud kernel image did not include the rdma_rxe module, so modprobe could not find the module; installing the additional module package (linux-modules-extra-<커널 버전>) brought it up. Since performance numbers are CPU-bound, this is strictly for verifying concepts.

• The warning that "software implementations have no meaning in numbers" was correct. On the same machine, TCP was 8 times faster than RDMA writes. The benefits of RDMA (bypassing the kernel and CPU) occur when hardware handles the transmission, but in a software implementation, the CPU works for every packet, making it disadvantageous instead. This test is for visually observing concepts such as verbs flow, GID, and MTU.
• Reproduced the "pinned memory limit" failure from the table in Section 7. As a regular user, if you lower ulimit -l and write a 1MiB message, it fails with Failed to create MR, Couldn't allocate MR at 64KiB and 1,024KiB (server side failed to create mr) and succeeds at 8,192KiB and above. A 4KiB message succeeds under the same 64KiB limit, meaning that what triggers the limit is the size of the memory to register. As root, it also succeeded at 64KiB. This is explained by the Linux kernel source (drivers/infiniband/core/umem.c) failing user memory registration when RLIMIT_MEMLOCK is exceeded CAP_IPC_LOCK only when there is no (in this test, capabilities were not checked directly, but verified via source descriptions).
• MTU is determined by the device. At Ethernet MTU 1500 active_mtu became 1024 (the rxe source code rounds down the Ethernet MTU minus the 80-byte header to the nearest power of 2 below, between 256 and 4096; 1500 minus 80 is 1420, which rounds down to 1024. The mlx5 device example in the NVIDIA documentation also shows active_mtu 1024 , so this is not a phenomenon unique to software implementations), when investigating issues where MTU mismatches occur between nodes, you must look at not only the Ethernet MTU but also ibv_devinfo 's active_mtu .
• The reason why there are multiple GIDs becomes clearly visible in practice. Because link-local, mapped IPv4, and addresses from other interfaces are brought up together on a single device, show_gids helps you concretely understand the explanation in Section 7 that you must select the term corresponding to RoCE v2 using . The behavior when the GID selection is incorrect was tested separately using ib_write_bw . When the server and client used the same index, all three types (IPv4-mapped, link-local, and overlay address) succeeded, and throughputs were similar at 1,266, 1,322, and 1,299 MB/s (accessed via self-address within a single VM). If the server and client used different indices (1 to 0, 1 to 3) or provided a non-existent index (9), it failed with exit code 1. Omitting the index defaulted to index 1 (IPv4-mapped). However, this test was conducted with perftest rather than NCCL, and because error messages could not be collected, it was not verified whether the ibv_modify_qp ... Invalid argument mentioned in the NCCL documentation occurs in the same situation (unmeasured).

The limitations of this test are a single virtual machine, a software implementation, and no GPUs. RDMA between two nodes, the throughput and latency of actual NICs, PFC and ECN, nccl-tests, and GPUDirect RDMA were not measured (unmeasured).

External Evidence: Findings verified through papers and official materials

RoCE congestion control, descriptions of NCCL and GPUDirect, and the performance of software RoCE, which could not be directly measured, were verified using primary sources. For external figures, because equipment and conditions varied, only the consistency of directions and structures was evaluated.

Figures from materials that could not be opened due to access restrictions (IEEE Network 2019 paper, Kernel Bugzilla 190951, SoftRDMA) were omitted.

11. Potential Interview Questions

1. Why use RDMA instead of TCP? Because the NIC directly reads and writes to fixed memory without kernel copies or CPU involvement, latency and CPU overhead are reduced. GPUDirect RDMA further eliminates traversing host memory.
2. How do you choose between InfiniBand and RoCE v2, and what are the risks of RoCE? IB offers simple configuration with dedicated fabrics and an SM, whereas RoCE uses existing Ethernet and L3 routing but requires aligning PFC, ECN, DSCP, and MTU all together. Misconfigurations manifest as tail latency or failures exclusively between nodes, which are verified via switch counters.
3. all_reduce_perf If busbw is half of the NIC line rate, where should you check first? ib_write_bw Check whether the NIC itself achieves line rate using nvidia-smi topo -m , inspect the GPU-NIC distance using NCCL_DEBUG=INFO , and verify the selected transport (NET/IB, GDRDMA usage) and rail configuration using
4. . What is the diagnostic sequence if multi-node training suddenly slows down?ibstat Check link degradation (ibdiagnetRate), port error counters, specific rail imbalance ( ), and in the case of RoCE, PFC and ECN counters, in that order. Adjust NCCL variables only when the hardware is verified to be healthy.
5. What is required for a Kubernetes pod to use RDMA? MOFED and the device plugin (Network Operator), pod resource requests for rdma/... , nvidia_peermem, memlock limits, Multus secondary interfaces or SR-IOV if needed, and placement that assigns a NIC close to the GPU.

12. Common Misconceptions

• "RDMA is InfiniBand." RoCE and AWS EFA also provide RDMA (Dynamo RDMA documentation).
• "NCCL environment variables apply to all GPU communication." They do not apply to KV transfers going through NIXL/UCX.
• "busbw is the reciprocal of processing time." It is algbw multiplied by a correction factor and is a metric meant to be compared against hardware limits.
• "Since we have NVLink, communication between servers is also fast." In standard 8-GPU servers, NVLink connects only within the server, with NVL72 being an exception.
• "The more tuning variables you set, the better." NCCL documentation does not recommend constant settings.

References (Actually Accessed)

• NCCL 2.32.3 Documentation (overview, usage/collectives, env, troubleshooting networking·performance_and_tuning·gpu_troubleshooting): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/
• https://github.com/NVIDIA/nccl, https://github.com/NVIDIA/nccl-tests (README, doc/PERFORMANCE.md), https://github.com/linux-rdma/perftest
• NCCL 2.4 Blog: https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/
• https://github.com/Mellanox/network-operator, https://github.com/Mellanox/k8s-rdma-shared-dev-plugin, https://github.com/k8snetworkplumbingwg/multus-cni, https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin
• Dynamo Installation & RDMA Documentation: https://github.com/ai-dynamo/dynamo (release/1.5.0 docs/fern/pages/kubernetes/installation)
• vLLM: https://docs.vllm.ai/en/latest/serving/parallelism_scaling/, https://docs.vllm.ai/en/latest/usage/troubleshooting/
• GPUDirect: https://docs.nvidia.com/cuda/gpudirect-rdma/, https://docs.nvidia.com/gpudirect-storage/overview-guide/index.html
• RoCE: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-44/Layer-1-and-Switch-Ports/Quality-of-Service/RDMA-over-Converged-Ethernet-RoCE/ , https://networking-docs.nvidia.com/onyxum/3.10.4800/rdma-over-converged-ethernet-roce , https://en.wikipedia.org/wiki/RDMA_over_Converged_Ethernet , https://docs.min.io/memkv/operate/rocev2/ , https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_infiniband_and_rdma_networks/configuring-roce_configuring-infiniband-and-rdma-networks
• Specs: https://www.nvidia.com/en-us/data-center/nvlink/ , https://docs.nvidia.com/dgx/dgxh100-user-guide/introduction-to-dgxh100.html , https://docs.nvidia.com/dgx-superpod/reference-architecture/scalable-infrastructure-rubinx86/latest/network-fabrics.html , https://www.rambus.com/blogs/pci-express-5-vs-4/
• Papers: https://arxiv.org/abs/2412.19437 , https://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p523.pdf (DCQCN, abstract level of search results only)

Checked versions, dates, and unverified items

The verification date is 2026-10-05. NCCL v2.32.3-1 (2026-09-17), perftest 26.07.8 (2026-09-28), Network Operator v26.4.2 (2026-09-16), RDMA Shared Device Plugin v1.5.4 (2026-07-28), Multus v4.3.1 (2026-09-08), and SR-IOV Device Plugin v3.11.0 (2025-12-16) were verified via the GitHub Releases API, while nccl-tests was read from master (commit of 2026-10-02) without a release.

Unverified items: (1) The main text of the DCQCN paper has now been read directly (see "External Evidence" below). (2) The system requirements in the Network Operator README (ConnectX-5, Ubuntu 20.04) appear outdated, and the current support matrix was not verified. (3) How to match and allocate GPU-NIC pairs in Kubernetes. (4) rdma link delete and the expected output of nccl-tests were not executed. (5) MTU negotiation rules are based on third-party runbooks. (6) InfiniBand link-layer flow control and NVLink 6th-generation details were not covered. (7) All three practical exercises in Section 10 remain unexecuted.