dspark cluster runs SINGLE-RAIL: NCCL_IB_HCA prefix excludes the second 200G rail (bosch<->escher) #9

New Issue

2026-07-22T16:51:46Z

williams commented

2026-07-22 16:51:46 +00:00

Finding

The bosch+escher dspark (vLLM) cluster is healthy and both nodes participate — but it uses only one of two available 200G rails.

Evidence

Both rails are up at full speed on both hosts:

	192.168.100.x	192.168.101.x
bosch	`enp1s0f1np1` .1 — 200000 Mb/s	`enP2p1s0f1np1` .3 — 200000 Mb/s
escher	`enp1s0f0np0` .2 — 200000 Mb/s	`enP2p1s0f0np0` .4 — 200000 Mb/s

But the container env filters NCCL to one of them:

NCCL_IB_HCA=rocep1s0f          <- matches rocep1s0f0/f1 ONLY
NCCL_SOCKET_IFNAME=enp1s0f     <- same problem
NCCL_CROSS_NIC=1
NCCL_NET=IB

ibdev2netdev on escher:

roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)   <- 192.168.101.x rail, NOT matched by the filter
rocep1s0f0   port 1 ==> enp1s0f0np0   (Up)   <- 192.168.100.x rail, in use
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down) <- expected (bosch uses f1)
rocep1s0f1   port 1 ==> enp1s0f1np1   (Down) <- expected

The second rail's HCA is roceP2p1s0f0 (capital P2p), so the rocep1s0f prefix cannot match it. A fully-up 200G link sits idle.

Why it matters

This matches the design note in the DGX rig memory: "full 200G needs bonding the two 100G MACs across different PCIe x4 links (same-link caps ~92-95 Gbps)". That is almost certainly the origin of the ~90 Gbps figure recorded in stash /projects — a single-rail measurement.

Fix

Widen the HCA filter on both nodes to include both rails, e.g.:

escher: NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
bosch: NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1

(and the matching NCCL_SOCKET_IFNAME). NCCL_CROSS_NIC=1 is already set favourably. Requires a coordinated restart of the dspark container on both nodes, which drops the serving DeepSeek endpoint — hence not applied live.

Measure before/after (perftest or an NCCL all-reduce bench) rather than assuming the change helps; PHY errors on lane 0/1 were previously noted.

Stale memory to correct

stash /projects says the link "works at ~90 Gbps", which is now doubly misleading: the rails negotiate 200G each, and the entry implies the link is being fully exploited when it is single-rail. Correct after measuring.

## Finding The bosch+escher dspark (vLLM) cluster is healthy and both nodes participate — but it uses **only one of two available 200G rails**. ## Evidence Both rails are up at full speed on both hosts: | | 192.168.100.x | 192.168.101.x | |---|---|---| | bosch | `enp1s0f1np1` .1 — **200000 Mb/s** | `enP2p1s0f1np1` .3 — **200000 Mb/s** | | escher | `enp1s0f0np0` .2 — **200000 Mb/s** | `enP2p1s0f0np0` .4 — **200000 Mb/s** | But the container env filters NCCL to one of them: ``` NCCL_IB_HCA=rocep1s0f <- matches rocep1s0f0/f1 ONLY NCCL_SOCKET_IFNAME=enp1s0f <- same problem NCCL_CROSS_NIC=1 NCCL_NET=IB ``` `ibdev2netdev` on escher: ``` roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up) <- 192.168.101.x rail, NOT matched by the filter rocep1s0f0 port 1 ==> enp1s0f0np0 (Up) <- 192.168.100.x rail, in use roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down) <- expected (bosch uses f1) rocep1s0f1 port 1 ==> enp1s0f1np1 (Down) <- expected ``` The second rail's HCA is `roceP2p1s0f0` (capital `P2p`), so the `rocep1s0f` prefix cannot match it. A fully-up 200G link sits idle. ## Why it matters This matches the design note in the DGX rig memory: *"full 200G needs bonding the two 100G MACs across different PCIe x4 links (same-link caps ~92-95 Gbps)"*. That is almost certainly the origin of the **~90 Gbps** figure recorded in stash `/projects` — a single-rail measurement. ## Fix Widen the HCA filter on **both** nodes to include both rails, e.g.: - escher: `NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0` - bosch: `NCCL_IB_HCA=rocep1s0f1,roceP2p1s0f1` (and the matching `NCCL_SOCKET_IFNAME`). `NCCL_CROSS_NIC=1` is already set favourably. Requires a **coordinated restart of the dspark container on both nodes**, which drops the serving DeepSeek endpoint — hence not applied live. Measure before/after (perftest or an NCCL all-reduce bench) rather than assuming the change helps; PHY errors on lane 0/1 were previously noted. ## Stale memory to correct stash `/projects` says the link *"works at ~90 Gbps"*, which is now doubly misleading: the rails negotiate 200G each, and the entry implies the link is being fully exploited when it is single-rail. Correct after measuring.

williams commented

2026-07-22 21:23:07 +00:00

Correction to the premise above (markus, verified by web research)

~200 Gbit/s is the TOTAL for a DGX Spark, not per rail. Both QSFP ports share the same two PCIe Gen5 x4 links to the GB10 SoC (ConnectX-7 in multi-host mode, because the Grace SoC only exposes x4 root ports). One PCIe Gen5 x4 is roughly 100 Gbps, so the ceiling is ~200 Gbps whether one or two cables are used. Real-world maximum is reported around 206 Gbps and needs IB/RDMA plus careful topology.

Sources: ServeTheHome — the GB10 ConnectX-7 200GbE networking is really different, DGX Spark QSFP DeepDive, NVIDIA Developer Forums.

What this changes

The original text implies a second, fully-idle 200G rail, i.e. ~400G available. That is wrong. The ceiling is ~200 Gbps total.
ethtool reporting 200000Mb/s on each interface is the negotiated cage speed, not achievable per-rail throughput.
The ~90 Gbps figure recorded in stash /projects is therefore NOT stale and should not be 'corrected' to 200 — it is an accurate single-rail measurement, sitting right at the ~92-95 Gbps ceiling of one PCIe Gen5 x4 link, exactly as the rig design note predicted.

What this does NOT change

The fix is still correct and still worth doing. NCCL_IB_HCA=rocep1s0f matches only one HCA, so the cluster currently rides a single PCIe x4 link and is capped near ~90-100 Gbps. Widening the filter to both rails should roughly double effective bandwidth toward the ~200 Gbps ceiling — it does not unlock 400G.

Expectation to measure against after the change: ~180-200 Gbps aggregate, not 2x200.

## Correction to the premise above (markus, verified by web research) **~200 Gbit/s is the TOTAL for a DGX Spark, not per rail.** Both QSFP ports share the *same two* PCIe Gen5 x4 links to the GB10 SoC (ConnectX-7 in multi-host mode, because the Grace SoC only exposes x4 root ports). One PCIe Gen5 x4 is roughly 100 Gbps, so the ceiling is ~200 Gbps **whether one or two cables are used**. Real-world maximum is reported around 206 Gbps and needs IB/RDMA plus careful topology. Sources: [ServeTheHome — the GB10 ConnectX-7 200GbE networking is really different](https://www.servethehome.com/the-nvidia-gb10-connectx-7-200gbe-networking-is-really-different/), [DGX Spark QSFP DeepDive](https://github.com/vroomfondel/dgxarley/blob/main/docs/DGX%20Spark%20QSFP%20DeepDive%20EN.md), [NVIDIA Developer Forums](https://forums.developer.nvidia.com/t/dgx-spark-200gbe-direct-qsfp-link-negotiates-200g-but-payload-is-12-gbps/373538). ### What this changes - The original text implies a second, fully-idle **200G** rail, i.e. ~400G available. **That is wrong.** The ceiling is ~200 Gbps total. - `ethtool` reporting `200000Mb/s` on each interface is the negotiated **cage** speed, not achievable per-rail throughput. - The **~90 Gbps** figure recorded in stash `/projects` is therefore **NOT stale and should not be 'corrected' to 200** — it is an accurate single-rail measurement, sitting right at the ~92-95 Gbps ceiling of one PCIe Gen5 x4 link, exactly as the rig design note predicted. ### What this does NOT change The fix is still correct and still worth doing. `NCCL_IB_HCA=rocep1s0f` matches only one HCA, so the cluster currently rides a single PCIe x4 link and is capped near ~90-100 Gbps. Widening the filter to both rails should roughly **double** effective bandwidth toward the ~200 Gbps ceiling — it does not unlock 400G. Expectation to measure against after the change: ~180-200 Gbps aggregate, not 2x200.

marfrit commented

2026-07-23 03:42:51 +00:00

Closing at Markus’ request — lack of information to act on this. The second 200G rail (roceP2p1s0f0) is up and idle, but we do not have the confirmed NCCL_IB_HCA value or a verified way to bring both rails into the collective without a coordinated restart that drops the serving DeepSeek endpoint. Reopen if the dual-rail config is characterized. Note for the record: the ~90 Gbps figure in stash /projects is the correct SINGLE-RAIL number (I twice wrongly called it stale); ~200G is the whole-box total across both PCIe Gen5 x4 links.

marfrit closed this issue

2026-07-23 03:42:51 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: marfrit/claude-his-agent#9