Distributed LLM Inference over RoCEv2
GB10 NVIDIA DGX Spark
How can a relatively affordable desktop platform support a compute cluster, and what challenges emerge when scaling a new architecture?
Daniel Mihovch
Can desktop-class nodes add useful model capacity?
Research question: can inter-node communication stay within interactive latency and goodput targets?
Why DGX Spark for an FP4 compute cluster?
GB10 provides 128 GB of unified memory in a compact desktop. The GB300 DGX Station is NVIDIA's larger deskside FP4 platform.
| Specification | DGX Spark · GB10 | DGX Station · GB300 |
|---|---|---|
| Unified memory | 128 GB | 748 GB |
| Memory bandwidth | 273 GB/s · LPDDR5X | 7.1 TB/s · GPU HBM3e 396 GB/s · CPU LPDDR5X |
| Peak sparse FP4 | Up to 1 PFLOP | Up to 20 PFLOPS |
| Physical format | 150 × 150 × 50.5 mm desktop | Workstation tower; OEM-dependent |
One Station provides 5.84× the coherent memory. US listed prices: Spark $6,950; BIZON GB300 Station from $132,159 (19.0×).
Specifications: NVIDIA Spark; NVIDIA Station. Prices checked November 9, 2026: NVIDIA US listing; BIZON starting configuration. Tax and shipping are not compared.
Two-node groundwork for 10, 20, and 30-node research
The two-node testbed evaluates communication limits and serving reliability before larger cluster studies.
This preliminary work establishes viability criteria for 10, 20, and 30-node research targets. Those topologies remain planned.
Component bandwidth sets the limits to investigate
Measure one interface, then all four together. Component ratings do not establish sustained application bandwidth.
Memory and port ratings: NVIDIA Spark specifications and networking guide. PCIe: retained kernel report, 126.028 Gb/s per ×4 link; combined value is derived. Rates are per direction.
Each Spark has two ports and four logical RoCEv2 devices
Two cables expose all four interfaces for network-heavy inter-node communication. They do not remove the shared host-link limit.
Research question: does using all four interfaces improve sustained bandwidth?
Hypothesis: No, but bonding inter-port interface pairs can provide redundancy
Topology: NVIDIA ConnectX-7 networking guide. Logical device numbers are schematic labels.
PerfTest-Suite: measures the fabric independently
I built the orchestration and evidence capture around the linux-rdma/perftest InfiniBand verbs microbenchmarks, run here over RoCEv2.
Deliver a message into a receive buffer posted by the peer.
Push bytes directly into the peer's registered memory.
Pull bytes directly from the peer's registered memory.
Isolated: one logical device runs at a time.
Concurrent: all four devices run the same test simultaneously.
The suite records peer mappings, commands, raw logs, CSV results, and link/PCIe metadata.
T (s) = L (s) + V (GB) / B (GB/s)
L: Smallest-payload typical latency.
B: Largest-payload average throughput.
V: Payload size.
PerfTest-Suite: large transfers approach a per-interface limit
Isolated interfaces exceed a 100 Gb/s application reference and remain below the ≈126 Gb/s PCIe host-link reference.
The plotted runs test one interface at a time. Four simultaneous rails are a separate condition.
The 100 Gb/s reference follows reported per-interface bandwidth results. Slide 13 shows aggregate NCCL bandwidth. Neither reference is a universal application ceiling.
PerfTest-Suite: short messages reveal baseline latency
µs
Small messages reveal the delay paid on each cross-node exchange.
This plot measures pairwise verbs operations, not end-to-end token latency.
PerfTest-Suite estimates fixed-volume transfer time
The suite uses the Hockney model, combining small-message latency with large-message bandwidth to estimate transfer time.
The estimate is a model of the data phase. It is not a measured process wall time.
The four panels retain each device's isolation and concurrent modes.
Tensor and pipeline parallelism communicate differently
Tensor parallelism
Layer weights remain sharded across nodes.
Collectives within each layer
Pipeline parallelism
Each node holds complete layers in one stage.
Activation transfer at stage boundaries
Conceptual communication pattern. TP can also use reduce-scatter/all-gather; the sequence depends on the model and implementation. NVIDIA Megatron Core reference.
11 / 25vLLM serves; Ray places workers; NCCL moves tensors
On Spark's unified-memory platform, GPUDirect RDMA is unavailable. RDMA communication uses registered host-memory buffers. NVIDIA porting guide.
Historical NCCL measurements compare sockets and RoCEv2
At 1 GiB, NCCL bus bandwidth: TCP sockets, all-reduce 2.12 GB/s and send/receive 3.05 GB/s; RoCEv2, all-reduce 21.18 GB/s and send/receive 24.09 GB/s.
Socket transport uses TCP/IP. RoCEv2 uses RDMA verbs over UDP/IP on Ethernet.
Historical measurements use different configurations and software revisions. A controlled comparison is required to quantify the transport contribution.
RoCEv2 pilot: serving methodology
Qwen3-Next-80B-A3B-Instruct · October 2026 pilot.
Synthetic prompts with controlled token lengths; seed 1234.
Request mix
| Input → output tokens | Weight |
|---|---|
| 128 → 128 | 1 |
| 1,024 → 256 | 1 |
| 4,096 → 256 | 1 |
| 8,192 → 512 | 0.5 |
Serving conditions
NVFP4: one Spark, TP=2, PP=2.
BF16: TP=2, PP=2.
NGC 26.04; vLLM 0.19.0 NVIDIA build.
Prefix caching disabled; context limit 8,704 tokens.
KV-cache settings: BF16 auto; NVFP4 fp8_e4m3.
Goodput targets: TTFT ≤ 3 s and request mean ITL ≤ 100 ms.
Closed-loop load; one run per point. Plotted concurrencies: 1, 2, 4, 8, 16, 32, 64. Source: retained pilot summaries and serving policies. Measurement-started requests are tracked to completion.
Qwen3-Next 80B-A3B: precision and model placement
Qwen3-Next 80B-A3B, comparing BF16 and NVFP4 in the retained RoCEv2 pilot.
Weight format, compute dtype, and KV-cache dtype are distinct. Model fit also depends on context and scheduler memory.
Sample GPU power with nvidia-smi; integrate over the measurement window, sum participating nodes, and divide by output tokens. Whole-system power is excluded.
vLLM memory utilization was set to 0.80 to leave headroom for other system activity.
Future tests will raise the memory budget as far as possible.
Serving quality is a request-level measurement
RoCEv2 pilot: first-token timing by concurrency
Qwen3-Next 80B NVFP4. All plotted median TTFT values stay below the 3 s target. Goodput still qualifies individual requests.
RoCEv2 pilot: inter-token timing by concurrency
Qwen3-Next 80B NVFP4. Each point is the median across request mean ITLs. All three configurations exceed the 100 ms target at concurrency 32.
RoCEv2 pilot: goodput depends on load and parallelism
In this pilot, goodput peaks at concurrency 16 and is zero at 32 and 64.
At concurrency 32, no requests meet both TTFT ≤ 3 s and mean ITL ≤ 100 ms. Requests continue completing. The cause of increased ITL remains unverified.
Target attainment falls to zero at concurrency 32
At concurrency 16, 98.2–100% of requests meet both targets. At 32 and 64, none qualify.
100 × requests meeting both targets / all requests started in the measurement window. Targets: TTFT ≤ 3 s and mean ITL ≤ 100 ms.
Hypothesis: memory bandwidth limits decoding
At concurrency 32, median TTFT stays below 3 s; median request mean ITL exceeds 100 ms.
Prediction
Sharding with parallel memory access may lower ITL if memory-access savings exceed added communication time.
Proposed test
Profile memory traffic, compute, scheduling, and communication at a fixed workload and total concurrency.
Untested hypothesis about memory bandwidth. Saturation and scale-out gains remain unmeasured. TTFT also includes queueing; its median does not establish the bottleneck. General model: NVIDIA inference phases; parallelism tradeoffs.
Scaling decisions require fabric, serving, and reliability evidence
Latency, bandwidth, transport selected, and concurrent rails.
TTFT, ITL, and goodput for a declared model and workload.
Logs, lifecycle state, recovery, and provenance.
Next test: bond interfaces for failover and compare bandwidth under heavy traffic to determine whether it changes.
Preliminary switched-pair results: higher latency
Untested hypothesis: Switch forwarding adds delay that could affect serving performance.
The topology contribution and serving impact remain unmeasured.
Next steps: validate the switched pair, then expand
Current switched pair, as reported by the research team.
Validate repeatability, fabric behavior, and serving reliability. If results are reliable, join the standalone pair to the switched pair.
Solid links: current pair. Dashed links: proposed expansion. The switch rating does not raise a Spark port's up-to-200-Gb/s rating.
Thank you
Questions?