DeepEP on AWS EFA: a measured guide to MoE expert-parallel all-to-all measured
DeepEP accelerates Mixture-of-Experts all-to-all by reducing communication volume and using
GPU-initiated networking. On InfiniBand + NVLink that is a real serving win. On AWS EFA it is not — and
this site shows, with the math and the measured numbers, exactly why, where DeepEP still has value, what the
alternatives are, and what would change the verdict. Built from a multi-week campaign: 3 frameworks served,
377 falsified hypotheses (ledger as of 2026-07-17), a fair A/B across regimes.
The one-paragraph answer
On AWS EFA (p5en, CPU-proxy NCCL-GIN), DeepEP-V2 is a working drop-in across SGLang, vLLM, and
TRT-LLM — all serve coherent MoE inference and pass AIPerf. But it does not beat a dense / allgather
alternative on serving throughput: measured DeepEP/alternative aggregate tok/s is 0.65–0.69×
on TRT-LLM and ~0.97× (a tie) on vLLM. The cause is structural — EFA's CPU-proxy path is
issue-bound at ~2.2% NIC utilization, paying a fixed +14.7 ms/token host round-trip that the
all-to-all byte-reduction can't overcome. Use DeepEP for capacity (fitting a model that won't otherwise
fit) or framework-fit — not for a serving speedup. That speedup needs GPU-initiated transport (GDAKI/IBGDA),
which EFA does not expose in any shipping stack today — though our §10 campaign has now measured a native kernel-posted research path that beats the proxy (dispatch 338.5 vs 391.6 µs) while staying ~2.5–2.9× off IB. On InfiniBand the calculus flips.
July 2026 update: all of this is now reproducible from public roots only (§9), and one
efficiency lever measurably converts — striping over 8 NCCL comms (EP_NUM_COMMS=8)
doubles vLLM serving throughput by attacking the issue-bound proxy directly. The
verdict direction is unchanged (dense still wins on TRT-LLM; capacity/fit is still the reason to use
DeepEP), but the gap halves.
The headline numbers (all measured)
0.65–0.69×DeepEP / dense agg tok/s on TRT-LLM, EP16, across conc4/conc32/ISL2048 measured
~0.97×DeepEP / allgather on vLLM decode — a statistical tie measured
+14.7 msfixed per-token CPU-proxy tax; load-invariant (per-user flat 19.6 vs 27.6 tok/s) derived
2.2%EFA NIC utilization on dispatch — issue-bound, the wire is 95%+ idle measured
1.45×where DeepEP DOES win: transport bytes (33 GB/s SO vs UCCL-published 22.77) measured
3 / 3frameworks serve DeepEP-V2 on EFA + pass AIPerf (SGLang, vLLM, TRT-LLM) measured
2×the multicomm lever (EP_NUM_COMMS=8) doubles vLLM on the PURE PUBLIC stack — the one lever that converts to serving tok/s. §9 measured
6 / 6gates reproducible from public roots only (micro + 3 engines + Megatron + NeMo-RL) — repo linked in §9 measured
Read in order
§1 · What is MoERouter, experts, dispatch/combine all-to-all, and why MoE needs expert parallelism.
open →
§2 · EFA vs InfiniBandThe fabric math: SRD vs RC, the proxy tax, why bytes ≠ throughput.
open →
§3 · ScalingSingle vs multi-node, conc/world-size/expert sweeps — charts.
open →
§4 · MicrobenchmarksTransport GB/s + real AIPerf inference numbers, sourced.
open →
§5 · Alternativesdense, UCCL, pplx, IBGDA — do they serve, how fast.
open →
§11 · FrameworksFramework support matrix:
3/3 serving (SGLang, vLLM, TRT-LLM pass AIPerf),
2/2 training (Megatron-LM, NeMo-RL GRPO E2E). Kimi-384 validation: 629.5 µs D+C micro-bench + ≤2.08e-4 MegatronBridge loss-match over model-dominated phase. Transport/dispatcher proven at production expert count, integration cost documented.
open →
§6 · Solution architectureWhat works today + internal/customer decision trees.
open →
§8 · DeepEP V1 (NVSHMEM backend)Campaign complete: four capstones (vLLM, SGLang, Megatron, NeMo-RL GRPO) measured 2026-06-25 → 2026-07-08 — serving + training + full RL over the same NVSHMEM/EFA transport. SGLang
inverts the vLLM result (LL wins via cuda-graph: 60–66 ms TPOT, decode c32 +50% tok/s). Megatron digit-for-digit numerics vs V2 (the only controlled cross-transport A/B). Integration cost: V1 needed
fewer patches (SGLang stock, Megatron 0 commits, NeMo-RL 1-hunk). H200 A/B (ours vs AWS, ~2× prefill gap), decode LL handle-cache bug + 3.7.0 fix.
Gap-closure retest: the vLLM LL-worse verdict was an eager artifact — cuda-graphed LL converges with SGLang (~60-65 ms) and edges dense 1.06-1.08× in decode.
open →
§9 · Reproduce it (public roots)The July 2026 re-run: all six gates from public upstreams + committed patches (
antonai-work/deepep-v2-efa-repro). NEW:
EP_NUM_COMMS=8 multicomm doubles vLLM (39.4/76.5/273.0 vs 19.5/38.8/144.6, ITL halved) — measured on the pure public stack. TRT-LLM native seam replaces the shim at zero cost.
open →
§10 · GPU-initiated (native, no proxy)Campaign 2026-07-09→17: GPU kernels post WQEs directly to the EFA NIC — correct (torch.equal, receiver-landing-proof) and
faster than the CPU proxy: dispatch 338.5 vs 391.6 µs, combine 480.7 vs 543.4 µs (decode, EP16). Honest residual: ~2.5–2.9× off the IB reference, attributed by experiment to EFA-SRD wire/readiness physics (73% fewer doorbells moved latency ~0; cached dispatch is 90% receiver-poll). Site verdict unchanged — the proxy stays the supported path.
open →
Try it on your config
The DeepEP-on-EFA Calculator ▸ takes your model,
EP degree, framework, and workload and returns a measured-calibrated verdict: DeepEP, dense, or capacity-forced.
This guidance IS
- Calibrated to measured fair A/B runs on real p5en EFA hardware.
- Specific to EFA / CPU-proxy NCCL-GIN (NCCL_GIN_TYPE=2, ElasticBuffer).
- Honest about where DeepEP wins (bytes, capacity, framework-fit) and where it doesn't (serving tok/s).
This guidance IS NOT
- A statement about InfiniBand/NVLink-only/MNNVL — there DeepEP+IBGDA can win.
- A correctness claim — both arms produce identical coherent output.
- A vendor benchmark — most serving cells are n=1; cross-regime ratio tightness is the variance evidence.