Distributed LLM Inference over RoCEv2
GB10 NVIDIA DGX Spark
Can desktop nodes add model capacity while keeping inter-node communication within interactive latency targets?
Daniel Mihovch
Can desktop-class nodes add useful model capacity?
Can inter-node communication stay within interactive latency and goodput targets?
Two-node pilot. Studies at 10, 20, and 30 nodes remain planned.
Component bandwidth sets the limits to investigate
Measure one interface, then all four together. Component ratings do not establish sustained application bandwidth.
Memory and port ratings: NVIDIA Spark specifications and networking guide. PCIe: retained kernel report, 126.028 Gb/s per ×4 link; combined value is derived. Rates are per direction.
Each Spark has two ports and four logical RoCEv2 devices
Two cables expose four logical paths. Shared host links still constrain sustained bandwidth.
Do four logical paths improve sustained bandwidth?
Topology: NVIDIA ConnectX-7 networking guide. Logical device numbers are schematic labels.
PerfTest-Suite: measures the fabric independently
I built the orchestration and evidence capture around the linux-rdma/perftest InfiniBand verbs microbenchmarks, run here over RoCEv2.
Deliver a message into a receive buffer posted by the peer.
Push bytes directly into the peer's registered memory.
Pull bytes directly from the peer's registered memory.
Isolated: one logical device at a time.
Concurrent: four clients launched together; verify successful overlap.
Capture peer mappings, commands, raw logs, CSV results, and link/PCIe metadata.
PerfTest-Suite: large transfers approach a per-interface limit
One-way payload bandwidth: ≈109 Gb/s per isolated interface.
Below the ≈126 Gb/s host-link reference. Concurrent capacity requires a separate test.
100 Gb/s: reported reference. Neither line is a universal ceiling. NCCL collective bandwidth is a separate metric.
PerfTest-Suite: short messages reveal baseline latency
µs
Small messages reveal the delay paid on each cross-node exchange.
This plot measures pairwise verbs operations, not end-to-end token latency.
Tensor and pipeline parallelism communicate differently
Tensor parallelism
Layer weights remain sharded across nodes.
Collectives within each layer
Pipeline parallelism
Each node holds complete layers in one stage.
Activation transfer at stage boundaries
Conceptual communication pattern. TP can also use reduce-scatter/all-gather; the sequence depends on the model and implementation. NVIDIA Megatron Core reference.
8 / 18vLLM serves; Ray places workers; NCCL moves tensors
On Spark's unified-memory platform, GPUDirect RDMA is unavailable. RDMA communication uses registered host-memory buffers. NVIDIA porting guide.
Historical NCCL measurements compare sockets and RoCEv2
At 1 GiB, NCCL bus bandwidth: TCP sockets, all-reduce 2.12 GB/s and send/receive 3.05 GB/s; RoCEv2, all-reduce 21.18 GB/s and send/receive 24.09 GB/s.
Socket transport uses TCP/IP. RoCEv2 uses RDMA verbs over UDP/IP on Ethernet.
Historical measurements use different configurations and software revisions. A controlled comparison is required to quantify the transport contribution.
RoCEv2 pilot: serving methodology
Qwen3-Next-80B-A3B-Instruct · NVFP4.
October 2026 pilot; synthetic prompts; seed 1234.
Request mix
| Input → output tokens | Weight |
|---|---|
| 128 → 128 | 1 |
| 1,024 → 256 | 1 |
| 4,096 → 256 | 1 |
| 8,192 → 512 | 0.5 |
Serving conditions
One Spark, TP=2, PP=2.
NGC 26.04; vLLM 0.19.0 NVIDIA build.
Prefix caching disabled; context limit 8,704 tokens.
KV cache: fp8_e4m3.
Goodput targets: TTFT ≤ 3 s and request mean ITL ≤ 100 ms.
Closed-loop load; one run per point. Plotted concurrencies: 1, 2, 4, 8, 16, 32, 64. Source: retained pilot summaries and serving policies. Measurement-started requests are tracked to completion.
Serving quality is a request-level measurement
RoCEv2 pilot: first-token timing by concurrency
Qwen3-Next 80B NVFP4. All plotted median TTFT values stay below the 3 s target. Goodput still qualifies individual requests.
RoCEv2 pilot: inter-token timing by concurrency
Qwen3-Next 80B NVFP4. Each point is the median across request mean ITLs. All three configurations exceed the 100 ms target at concurrency 32.
RoCEv2 pilot: goodput depends on load and parallelism
TP=2 peaks at 0.840 qualifying requests/s at concurrency 16.
98.2–100% of measurement-started requests qualify at concurrency 16; 0% at 32 and 64. Requests continue completing. The bottleneck remains unverified.
Preliminary switched-pair results: higher latency
Untested hypothesis: Switch forwarding adds delay that could affect serving performance.
The topology contribution and serving impact remain unmeasured.
Next steps: validate network costs before expanding
Current switched pair, as reported by the research team.
Repeat the direct/switch comparison at fixed conditions and verify NCCL transport. Expand after repeatable serving results.
Untested hypothesis: parallel memory access lowers ITL when memory-access savings exceed communication costs. Profile memory, compute, and communication to test it.
Solid: current pair. Dashed: proposed expansion. Spark ports remain rated up to 200 Gb/s. Memory/communication tradeoff.
Thank you
Questions?