Distributed LLM Inference over RoCEv2
GB10 NVIDIA DGX Spark

Can desktop nodes add model capacity while keeping inter-node communication within interactive latency targets?

Daniel Mihovch

Two DGX Spark systems connected by network cables in the experimental setup.
1 / 18

Can desktop-class nodes add useful model capacity?

One desktop system Model state must fit locally GB10 · 128 GB unified memory scale out Two desktop systems Spark A Spark B model shard A model shard B RoCEv2 carries synchronization and activations Local memory boundary Distributed placement adds network traffic

Can inter-node communication stay within interactive latency and goodput targets?

Two-node pilot. Studies at 10, 20, and 30 nodes remain planned.

2 / 18

Component bandwidth sets the limits to investigate

GB10 SoC Unified memory 273 GB/s memory bandwidth PCIe 5.0 ×4 ≈126 Gb/s PCIe 5.0 ×4 ≈126 Gb/s ConnectX-7 Spark NIC path 2 × 200 Gb/s external port ratings QSFP port 0 Up to 200 Gb/s QSFP port 1 Up to 200 Gb/s Two host links: ≈252 Gb/s combined before transaction overhead

Measure one interface, then all four together. Component ratings do not establish sustained application bandwidth.

Memory and port ratings: NVIDIA Spark specifications and networking guide. PCIe: retained kernel report, 126.028 Gb/s per ×4 link; combined value is derived. Rates are per direction.

3 / 18

Each Spark has two ports and four logical RoCEv2 devices

Head · Spark A Worker · Spark B 2 ports · 4 RoCEv2 devices 2 ports · 4 RoCEv2 devices Port 0 Port 0 Port 1 Port 1 logical devices 1 + 2 logical devices 1 + 2 logical devices 3 + 4 logical devices 3 + 4 Cable 1 · two logical paths Cable 2 · two logical paths

Two cables expose four logical paths. Shared host links still constrain sustained bandwidth.

Do four logical paths improve sustained bandwidth?

Topology: NVIDIA ConnectX-7 networking guide. Logical device numbers are schematic labels.

4 / 18

PerfTest-Suite: measures the fabric independently

I built the orchestration and evidence capture around the linux-rdma/perftest InfiniBand verbs microbenchmarks, run here over RoCEv2.

Send

Deliver a message into a receive buffer posted by the peer.

RDMA write

Push bytes directly into the peer's registered memory.

RDMA read

Pull bytes directly from the peer's registered memory.

Isolated: one logical device at a time.

Concurrent: four clients launched together; verify successful overlap.

Capture peer mappings, commands, raw logs, CSV results, and link/PCIe metadata.

5 / 18

PerfTest-Suite: large transfers approach a per-interface limit

Six-bar PerfTest chart: per-direction bandwidth in GB/s for send, write, and read, with one-way and bidirectional measurements and reference lines at 126 Gb/s and 100 Gb/s

One-way payload bandwidth: ≈109 Gb/s per isolated interface.

Below the ≈126 Gb/s host-link reference. Concurrent capacity requires a separate test.

100 Gb/s: reported reference. Neither line is a universal ceiling. NCCL collective bandwidth is a separate metric.

6 / 18

PerfTest-Suite: short messages reveal baseline latency

Original PerfTest latency plot comparing send, write, read, and atomic operations across min, typical, mean, p99, and p99.9

µs

Small messages reveal the delay paid on each cross-node exchange.

This plot measures pairwise verbs operations, not end-to-end token latency.

7 / 18

Tensor and pipeline parallelism communicate differently

Tensor parallelism

Layer weights remain sharded across nodes.

Node A Node B Layer 1 · A Layer 1 · B Layer 2 · A Layer 2 · B Layer 3 · A Layer 3 · B All-reduce activations All-reduce activations All-reduce activations

Collectives within each layer

Pipeline parallelism

Each node holds complete layers in one stage.

Node A Layers 1–3 Node B Layers 4–6 Send / receive activations Output → next decoding step

Activation transfer at stage boundaries

Conceptual communication pattern. TP can also use reduce-scatter/all-gather; the sequence depends on the model and implementation. NVIDIA Megatron Core reference.

8 / 18

vLLM serves; Ray places workers; NCCL moves tensors

Head node · Docker Worker node · Docker Client vLLM API / scheduler request Ray Placement / lifecycle vLLM GPU worker vLLM GPU worker GB10 · NCCL GB10 · NCCL placeworkers Tensor data NCCL overRoCEv2
Dashed: placement / lifecycle Solid: request / tensor data

On Spark's unified-memory platform, GPUDirect RDMA is unavailable. RDMA communication uses registered host-memory buffers. NVIDIA porting guide.

9 / 18

Historical NCCL measurements compare sockets and RoCEv2

Original repository plot of historical NCCL bus bandwidth versus message size for TCP sockets and RoCEv2 RDMA, with tensor all-reduce and pipeline send-receive curves

At 1 GiB, NCCL bus bandwidth: TCP sockets, all-reduce 2.12 GB/s and send/receive 3.05 GB/s; RoCEv2, all-reduce 21.18 GB/s and send/receive 24.09 GB/s.

Socket transport uses TCP/IP. RoCEv2 uses RDMA verbs over UDP/IP on Ethernet.

Historical measurements use different configurations and software revisions. A controlled comparison is required to quantify the transport contribution.

10 / 18

RoCEv2 pilot: serving methodology

Qwen3-Next-80B-A3B-Instruct · NVFP4.
October 2026 pilot; synthetic prompts; seed 1234.

Request mix

Input → output tokensWeight
128 → 1281
1,024 → 2561
4,096 → 2561
8,192 → 5120.5

Serving conditions

One Spark, TP=2, PP=2.

NGC 26.04; vLLM 0.19.0 NVIDIA build.
Prefix caching disabled; context limit 8,704 tokens.

KV cache: fp8_e4m3.

Goodput targets: TTFT ≤ 3 s and request mean ITL ≤ 100 ms.

≥120 sWarmup
1,500 sMeasurement window
≥180 sCooldown under load

Closed-loop load; one run per point. Plotted concurrencies: 1, 2, 4, 8, 16, 32, 64. Source: retained pilot summaries and serving policies. Measurement-started requests are tracked to completion.

11 / 18

Serving quality is a request-level measurement

Request Token 1 Token 2 Token 3 TTFT ≤ 3 s mean ITL ≤ 100 ms Goodput = requests meeting both targets / measured seconds
12 / 18

RoCEv2 pilot: first-token timing by concurrency

Original RoCEv2 pilot time to first token versus concurrency chart for Qwen3-Next 80B NVFP4
● One Spark ■ TP=2 ▲ PP=2

Qwen3-Next 80B NVFP4. All plotted median TTFT values stay below the 3 s target. Goodput still qualifies individual requests.

13 / 18

RoCEv2 pilot: inter-token timing by concurrency

Median request mean inter-token latency versus concurrency for Qwen3-Next 80B NVFP4. All three configurations cross the 100 millisecond request-mean target at concurrency 32.
● One Spark ■ TP=2 ▲ PP=2

Qwen3-Next 80B NVFP4. Each point is the median across request mean ITLs. All three configurations exceed the 100 ms target at concurrency 32.

14 / 18

RoCEv2 pilot: goodput depends on load and parallelism

Original RoCEv2 pilot goodput versus concurrent requests plot for Qwen3-Next 80B NVFP4 across single, tensor, and pipeline configurations
● One Spark ■ TP=2 ▲ PP=2

TP=2 peaks at 0.840 qualifying requests/s at concurrency 16.

98.2–100% of measurement-started requests qualify at concurrency 16; 0% at 32 and 64. Requests continue completing. The bottleneck remains unverified.

15 / 18

Preliminary switched-pair results: higher latency

PerfTest comparison of October 8 direct links and November 5 switched links. At 2 bytes, typical send, write, and read latency increases by 70 to 80 percent. At 64 KiB, isolated unidirectional bandwidth changes by less than 0.2 percent.

Untested hypothesis: Switch forwarding adds delay that could affect serving performance.

The topology contribution and serving impact remain unmeasured.

16 / 18

Next steps: validate network costs before expanding

Two DGX Spark systems with network interconnects above the network switch.

Current switched pair, as reported by the research team.

Switched Spark A current Switched Spark B current 400 Gb/s network switch Standalone Spark Standalone Spark proposed addition proposed addition

Repeat the direct/switch comparison at fixed conditions and verify NCCL transport. Expand after repeatable serving results.

Untested hypothesis: parallel memory access lowers ITL when memory-access savings exceed communication costs. Profile memory, compute, and communication to test it.

Solid: current pair. Dashed: proposed expansion. Spark ports remain rated up to 200 Gb/s. Memory/communication tradeoff.

17 / 18
18 / 18

Thank you

Questions?