Distributed LLM Inference over RoCEv2
GB10 NVIDIA DGX Spark

How can a relatively affordable desktop platform support a compute cluster, and what challenges emerge when scaling a new architecture?

Daniel Mihovch

Two DGX Spark systems connected by network cables in the experimental setup.
1 / 25

Can desktop-class nodes add useful model capacity?

One desktop system Model state must fit locally GB10 · 128 GB unified memory scale out Two desktop systems Spark A Spark B model shard A model shard B RoCEv2 carries synchronization and activations Local memory boundary Distributed placement adds network traffic

Research question: can inter-node communication stay within interactive latency and goodput targets?

2 / 25

Why DGX Spark for an FP4 compute cluster?

GB10 provides 128 GB of unified memory in a compact desktop. The GB300 DGX Station is NVIDIA's larger deskside FP4 platform.

Specification DGX Spark · GB10 DGX Station · GB300
Unified memory 128 GB 748 GB
Memory bandwidth 273 GB/s · LPDDR5X 7.1 TB/s · GPU HBM3e
396 GB/s · CPU LPDDR5X
Peak sparse FP4 Up to 1 PFLOP Up to 20 PFLOPS
Physical format 150 × 150 × 50.5 mm desktop Workstation tower; OEM-dependent

One Station provides 5.84× the coherent memory. US listed prices: Spark $6,950; BIZON GB300 Station from $132,159 (19.0×).

Specifications: NVIDIA Spark; NVIDIA Station. Prices checked November 9, 2026: NVIDIA US listing; BIZON starting configuration. Tax and shipping are not compared.

3 / 25

Two-node groundwork for 10, 20, and 30-node research

Preliminary testbed 2 nodes Research target 10 · 20 · 30 desktop nodes planned scale-out studies

The two-node testbed evaluates communication limits and serving reliability before larger cluster studies.

This preliminary work establishes viability criteria for 10, 20, and 30-node research targets. Those topologies remain planned.

4 / 25

Component bandwidth sets the limits to investigate

GB10 SoC Unified memory 273 GB/s memory bandwidth PCIe 5.0 ×4 ≈126 Gb/s PCIe 5.0 ×4 ≈126 Gb/s ConnectX-7 Spark NIC path 2 × 200 Gb/s external port ratings QSFP port 0 Up to 200 Gb/s QSFP port 1 Up to 200 Gb/s Two host links: ≈252 Gb/s combined before transaction overhead

Measure one interface, then all four together. Component ratings do not establish sustained application bandwidth.

Memory and port ratings: NVIDIA Spark specifications and networking guide. PCIe: retained kernel report, 126.028 Gb/s per ×4 link; combined value is derived. Rates are per direction.

5 / 25

Each Spark has two ports and four logical RoCEv2 devices

Head · Spark A Worker · Spark B 2 ports · 4 RoCEv2 devices 2 ports · 4 RoCEv2 devices Port 0 Port 0 Port 1 Port 1 logical devices 1 + 2 logical devices 1 + 2 logical devices 3 + 4 logical devices 3 + 4 Cable 1 · two logical paths Cable 2 · two logical paths

Two cables expose all four interfaces for network-heavy inter-node communication. They do not remove the shared host-link limit.

Research question: does using all four interfaces improve sustained bandwidth?

Hypothesis: No, but bonding inter-port interface pairs can provide redundancy

Topology: NVIDIA ConnectX-7 networking guide. Logical device numbers are schematic labels.

6 / 25

PerfTest-Suite: measures the fabric independently

I built the orchestration and evidence capture around the linux-rdma/perftest InfiniBand verbs microbenchmarks, run here over RoCEv2.

Send

Deliver a message into a receive buffer posted by the peer.

RDMA write

Push bytes directly into the peer's registered memory.

RDMA read

Pull bytes directly from the peer's registered memory.

Isolated: one logical device runs at a time.

Concurrent: all four devices run the same test simultaneously.

The suite records peer mappings, commands, raw logs, CSV results, and link/PCIe metadata.

Estimated transmission time

T (s) = L (s) + V (GB) / B (GB/s)

L: Smallest-payload typical latency.
B: Largest-payload average throughput.
V: Payload size.

7 / 25

PerfTest-Suite: large transfers approach a per-interface limit

Six-bar PerfTest chart: per-direction bandwidth in GB/s for send, write, and read, with one-way and bidirectional measurements and reference lines at 126 Gb/s and 100 Gb/s

Isolated interfaces exceed a 100 Gb/s application reference and remain below the ≈126 Gb/s PCIe host-link reference.

The plotted runs test one interface at a time. Four simultaneous rails are a separate condition.

The 100 Gb/s reference follows reported per-interface bandwidth results. Slide 13 shows aggregate NCCL bandwidth. Neither reference is a universal application ceiling.

8 / 25

PerfTest-Suite: short messages reveal baseline latency

Original PerfTest latency plot comparing send, write, read, and atomic operations across min, typical, mean, p99, and p99.9

µs

Small messages reveal the delay paid on each cross-node exchange.

This plot measures pairwise verbs operations, not end-to-end token latency.

9 / 25

PerfTest-Suite estimates fixed-volume transfer time

Original four-panel PerfTest write estimated transfer time plot for each RoCEv2 device over 0.1 to 100 GB volumes

The suite uses the Hockney model, combining small-message latency with large-message bandwidth to estimate transfer time.

The estimate is a model of the data phase. It is not a measured process wall time.

The four panels retain each device's isolation and concurrent modes.

10 / 25

Tensor and pipeline parallelism communicate differently

Tensor parallelism

Layer weights remain sharded across nodes.

Node A Node B Layer 1 · A Layer 1 · B Layer 2 · A Layer 2 · B Layer 3 · A Layer 3 · B All-reduce activations All-reduce activations All-reduce activations

Collectives within each layer

Pipeline parallelism

Each node holds complete layers in one stage.

Node A Layers 1–3 Node B Layers 4–6 Send / receive activations Output → next decoding step

Activation transfer at stage boundaries

Conceptual communication pattern. TP can also use reduce-scatter/all-gather; the sequence depends on the model and implementation. NVIDIA Megatron Core reference.

11 / 25

vLLM serves; Ray places workers; NCCL moves tensors

Head node · Docker Worker node · Docker Client vLLM API / scheduler request Ray Placement / lifecycle vLLM GPU worker vLLM GPU worker GB10 · NCCL GB10 · NCCL placeworkers Tensor data NCCL overRoCEv2
Dashed: placement / lifecycle Solid: request / tensor data

On Spark's unified-memory platform, GPUDirect RDMA is unavailable. RDMA communication uses registered host-memory buffers. NVIDIA porting guide.

12 / 25

Historical NCCL measurements compare sockets and RoCEv2

Original repository plot of historical NCCL bus bandwidth versus message size for TCP sockets and RoCEv2 RDMA, with tensor all-reduce and pipeline send-receive curves

At 1 GiB, NCCL bus bandwidth: TCP sockets, all-reduce 2.12 GB/s and send/receive 3.05 GB/s; RoCEv2, all-reduce 21.18 GB/s and send/receive 24.09 GB/s.

Socket transport uses TCP/IP. RoCEv2 uses RDMA verbs over UDP/IP on Ethernet.

Historical measurements use different configurations and software revisions. A controlled comparison is required to quantify the transport contribution.

13 / 25

RoCEv2 pilot: serving methodology

Qwen3-Next-80B-A3B-Instruct · October 2026 pilot.
Synthetic prompts with controlled token lengths; seed 1234.

Request mix

Input → output tokensWeight
128 → 1281
1,024 → 2561
4,096 → 2561
8,192 → 5120.5

Serving conditions

NVFP4: one Spark, TP=2, PP=2.
BF16: TP=2, PP=2.

NGC 26.04; vLLM 0.19.0 NVIDIA build.
Prefix caching disabled; context limit 8,704 tokens.

KV-cache settings: BF16 auto; NVFP4 fp8_e4m3.

Goodput targets: TTFT ≤ 3 s and request mean ITL ≤ 100 ms.

≥120 sWarmup
1,500 sMeasurement window
≥180 sCooldown under load

Closed-loop load; one run per point. Plotted concurrencies: 1, 2, 4, 8, 16, 32, 64. Source: retained pilot summaries and serving policies. Measurement-started requests are tracked to completion.

14 / 25

Qwen3-Next 80B-A3B: precision and model placement

Qwen3-Next 80B BF16 and NVFP4 output throughput and GPU energy per token by parallelism through 64 concurrent requests

Qwen3-Next 80B-A3B, comparing BF16 and NVFP4 in the retained RoCEv2 pilot.

Weight format, compute dtype, and KV-cache dtype are distinct. Model fit also depends on context and scheduler memory.

Sample GPU power with nvidia-smi; integrate over the measurement window, sum participating nodes, and divide by output tokens. Whole-system power is excluded.

vLLM memory utilization was set to 0.80 to leave headroom for other system activity.

Future tests will raise the memory budget as far as possible.

15 / 25

Serving quality is a request-level measurement

Request Token 1 Token 2 Token 3 TTFT ≤ 3 s mean ITL ≤ 100 ms Goodput = requests meeting both targets / measured seconds
16 / 25

RoCEv2 pilot: first-token timing by concurrency

Original RoCEv2 pilot time to first token versus concurrency chart for Qwen3-Next 80B NVFP4
● One Spark ■ TP=2 ▲ PP=2

Qwen3-Next 80B NVFP4. All plotted median TTFT values stay below the 3 s target. Goodput still qualifies individual requests.

17 / 25

RoCEv2 pilot: inter-token timing by concurrency

Median request mean inter-token latency versus concurrency for Qwen3-Next 80B NVFP4. All three configurations cross the 100 millisecond request-mean target at concurrency 32.
● One Spark ■ TP=2 ▲ PP=2

Qwen3-Next 80B NVFP4. Each point is the median across request mean ITLs. All three configurations exceed the 100 ms target at concurrency 32.

18 / 25

RoCEv2 pilot: goodput depends on load and parallelism

Original RoCEv2 pilot goodput versus concurrent requests plot for Qwen3-Next 80B NVFP4 across single, tensor, and pipeline configurations
● One Spark ■ TP=2 ▲ PP=2

In this pilot, goodput peaks at concurrency 16 and is zero at 32 and 64.

At concurrency 32, no requests meet both TTFT ≤ 3 s and mean ITL ≤ 100 ms. Requests continue completing. The cause of increased ITL remains unverified.

19 / 25

Target attainment falls to zero at concurrency 32

Percentage of Qwen3-Next 80B NVFP4 requests meeting both latency targets versus concurrency. At concurrency 16, 98.2 to 100 percent qualify; at 32 and 64, zero percent qualify in every configuration.
● One Spark ■ TP=2 ▲ PP=2

At concurrency 16, 98.2–100% of requests meet both targets. At 32 and 64, none qualify.

100 × requests meeting both targets / all requests started in the measurement window. Targets: TTFT ≤ 3 s and mean ITL ≤ 100 ms.

20 / 25

Hypothesis: memory bandwidth limits decoding

At concurrency 32, median TTFT stays below 3 s; median request mean ITL exceeds 100 ms.

Prefill Prompt tokens processed in parallel Often compute-bound Contributes to TTFT Decode Repeated weight and state access Can be memory-bandwidth bound Generates subsequent tokens; affects ITL First token

Prediction

Sharding with parallel memory access may lower ITL if memory-access savings exceed added communication time.

Proposed test

Profile memory traffic, compute, scheduling, and communication at a fixed workload and total concurrency.

Untested hypothesis about memory bandwidth. Saturation and scale-out gains remain unmeasured. TTFT also includes queueing; its median does not establish the bottleneck. General model: NVIDIA inference phases; parallelism tradeoffs.

21 / 25

Scaling decisions require fabric, serving, and reliability evidence

Fabric

Latency, bandwidth, transport selected, and concurrent rails.

Serving

TTFT, ITL, and goodput for a declared model and workload.

Reliability

Logs, lifecycle state, recovery, and provenance.

Evidence for a scale decision

Next test: bond interfaces for failover and compare bandwidth under heavy traffic to determine whether it changes.

22 / 25

Preliminary switched-pair results: higher latency

PerfTest comparison of October 8 direct links and November 5 switched links. At 2 bytes, typical send, write, and read latency increases by 70 to 80 percent. At 64 KiB, isolated unidirectional bandwidth changes by less than 0.2 percent.

Untested hypothesis: Switch forwarding adds delay that could affect serving performance.

The topology contribution and serving impact remain unmeasured.

23 / 25

Next steps: validate the switched pair, then expand

Two DGX Spark systems with network interconnects above the network switch.

Current switched pair, as reported by the research team.

Switched Spark A current Switched Spark B current 400 Gb/s network switch Standalone Spark Standalone Spark proposed addition proposed addition

Validate repeatability, fabric behavior, and serving reliability. If results are reliable, join the standalone pair to the switched pair.

Solid links: current pair. Dashed links: proposed expansion. The switch rating does not raise a Spark port's up-to-200-Gb/s rating.

24 / 25
25 / 25

Thank you

Questions?