DeepEP on EFA

§10 · GPU-initiated on EFA — the native path, measured against the proxy measured

Every earlier section runs DeepEP-V2 through the CPU-proxy GIN backend: a host thread posts the RDMA work. This campaign built the path everyone asks about — GPU kernels posting WQEs directly to the EFA NIC (kernel-posted-WQE, CQ-poll completion, no hardware counters needed) — got it correct, and then spent a week of one-lever-at-a-time optimization finding out exactly where the remaining microseconds live. The result is a first: native GPU-initiated beats the CPU proxy on both dispatch and combine at decode shape on EFA. The result is also honest: it does not reach InfiniBand-class latency, and we can now say why with evidence rather than intuition.

338.5 µs
native dispatch p50 (proxy 391.6)
480.7 µs
native combine p50 (proxy 543.4)
100%
torch.equal vs NCCL ref, ≥2 reps
377
falsified-hypothesis ledger entries

The headline table

2× p5en.48xlarge (H200, 16 EFA NICs/node) · 16 EP ranks · N=128 decode shape · p50 across valid reps · banked 2026-07-14, unchanged by the 2026-07-17 re-bank
axisnative GPU-initiatedCPU proxyverdict
dispatch338.5 µs391.6 µsnative −53 µs (~14%)
combine480.7 µs543.4 µsnative −63 µs (~12%)
cached dispatch~401 µs~274 µsproxy wins (see attribution)

source: gda-native-campaign/results/campaign-summary.json in the evidence repo. Correctness gate: dispatch+combine torch.equal == NCCL reference with receiver-side landing proof — local send-CQ success was never accepted as delivery.

How it got there — the lever ladder

The first valid A/B (2026-07-10) had the native path 2× slower than the proxy. One-variable-at-a-time optimization, each lever preregistered with validate/refute bands and adjudicated by an independent reviewer:

each row = one gated experiment with custody (preflight, source hashes, rank rc, hygiene)
leveroutcome
QP fanout fix (stop reserving a notify-only QP when SMs need data QPs)dispatch 789 → 538 µs
QP17 production shape + ownershipvalidated baseline
lazy completion on dispatch (skip per-put credit check when SQ overflow provably impossible)−30 µs; completion-wait → 0%
lazy completion on combine (C-1)−91 µs combine
batched doorbells (B=8)refuted as dominant (−2.8 µs)
write-with-immediate receiver countingfalsified: +382 µs sender-WQE tax
provider batched-tailcorrect but +198 µs
multi-WQE span posting (put_many)parked: deterministic hang, codegen-fragile (5 fix classes falsified, SASS forensics banked)
deferred-doorbell aggregation (73% fewer doorbell+fence events)engaged, latency FLAT — then intermittent correctness fault; parked

Why the IB gap stays — three measured facts

1. Completion-wait is gone. After lazy completion, profiled combine put-path completion-wait is 0.00% over 15.6k sampled rows (it was 38.6% before — the same instrument measured both).

2. Issue-side cost is overlapped. The aggregation experiment cut doorbell+fence events by 73% — and combine p50 moved ~0 µs. The ring/fence cycles the profiler attributes (72% of put cost) are hidden under wire time. Optimizing them buys nothing.

3. Cached dispatch is receiver physics. The one axis the proxy still wins is 89.96% receiver-forward poll wait (8.96× wait/body) — waiting for tokens to land and become visible, not posting them.

Against the DeepEP-V2 IB/IBGDA reference (~118/195 µs), native EFA remains ~2.5–2.9×. The residual is EFA-SRD wire and readiness behavior (no ordering, receiver-visibility latency), not host or issue-side code.

What this changes — and what it doesn't

it IS

  • proof GPU-initiated DeepEP-V2 dispatch/combine is correct on EFA (no hardware counters needed — CQ-poll completion)
  • a measured native win over the proxy on dispatch and combine at decode shape
  • an evidence-backed attribution of every remaining microsecond class

it is NOT

  • a production path — it needs a patched plugin/NCCL stack; the proxy (§3–§9) remains the supported recipe
  • an IB-parity claim — the ~2.5–2.9× reference gap stands, now with physics-level evidence
  • a change to the site verdict: DeepEP on EFA is a capacity/fit play; this moves the research frontier within it
Campaign 2026-07-09 → 2026-07-17 · 2× p5en.48xlarge (H200, 16 EFA NICs/node) · DeepEP-V2 type-5 EFA-GDA vs NCCL_GIN_TYPE=2 proxy · 16 EP ranks, N=128 decode · every number traces to deepep-v2-efa-repro/gda-native-campaign · Sanitized — no internal IPs, registry IDs, or hostnames.