DeepEP is one way to do Mixture-of-Experts all-to-all. There are five paths a serving stack can take on AWS EFA today. This section surveys each one with the same four questions: what is it, does inference actually serve on EFA (yes / no / with-what), what did we measure, and when would you pick it. Everything tagged measured ran on the efa-gda campaign — 2× p5en.48xlarge, 16 EFA NICs/node, Qwen3-30B-A3B-FP8 EP16 unless noted — not vendor spec sheets.
Rows are the all-to-all alternatives. Read left-to-right: does it serve on EFA, how fast is the transport, how fast is end-to-end serving, does it need a code fork, and the one-line "pick it when". Green = yes / fast / no-fork. Amber = ties / caveat. Red = no / slow / blocker.
Verdict colours: ● yes / fast / no-fork ● ties / caveat / partial ● no / blocker / fork-required. "transport speed" = dispatch/combine microbench (bytes-on-the-wire); "serving speed" = end-to-end aggregate tok/s. The two columns disagree on purpose — that disagreement is the whole story of this section.
Skip the expert-parallel all-to-all entirely. Each rank gathers every token
(allgather), every rank runs every expert it owns, then a reduce-scatter
returns the results. O(N) communication volume per GPU instead of DeepEP's O(top-k) — but it rides
NVLink intra-node and a small EFA collective cross-node, with no special transport stack.
In the frameworks: TRTLLM_MOE_DISABLE_ALLTOALLV (TRT-LLM) or
--all2all-backend allgather_reducescatter (vLLM).
Yes — everywhere, no caveats. It is the path that needs nothing: no GIN plugin, no ElasticBuffer, no QP-cap tuning, no proxy. If a framework runs on the box, this runs.
This is the default recommendation for EP≤16 serving on EFA. Reach for DeepEP only when a measurement or a capacity constraint says you should — not by default.
| measured A/B (same framework, same model/nodes, EP16) | DeepEP agg tok/s | dense agg tok/s | DeepEP / dense |
|---|---|---|---|
| TRT-LLM 0.21 · conc4 / ISL120 | 68.7 | 105.66 | 0.65× |
| TRT-LLM 0.21 · conc32 / ISL120 | 545.04 | 794.56 | 0.69× |
| vLLM · conc32 / decode | 296 | 304 | 0.97× (tie) |
| vLLM · conc64 / prefill | 18201 | 22797 | 0.80× |
measured Dense is 1.5× faster than DeepEP on TRT-LLM and ties DeepEP on vLLM. There is no measured serving regime on EFA where DeepEP beats dense — best case is a tie. That single fact is why dense is the default.
DeepSeek's expert-parallel all-to-all, V2 generation. On EFA it runs through
deep_ep.ElasticBuffer over the NCCL-GIN CPU-proxy path
(NCCL_GIN_TYPE=2) — not the IBGDA/NVSHMEM path the upstream
deep_ep.Buffer assumes, which EFA cannot run (no atomics, no ordering on SRD). The
proxy posts each cross-node dispatch via the host: D2H ring → CPU proxy → libfabric
fi_writedata → EFA SRD.
Yes — proven on all three frameworks. SGLang, vLLM and TRT-LLM all serve
DeepEP-V2 on EFA and pass AIPerf with coherent generation. The platform requirement: frameworks
must construct ElasticBuffer, not Buffer — asserted in the harness so a
silent fall-back to a broken transport cannot pass unnoticed.
EP_NUM_COMMS=8 (multi-comm)A single NCCL-GIN comm is issue-bound at ~8 GB/s SO — the CPU-proxy posts one
fi_writedata/doorbell per token, serialized on one progress thread. Minting K independent
comms behind the ElasticBuffer wall gives K progress threads and K× the issue ceiling.
EP_NUM_COMMS mints the comms from a
stock import.A raw-ibverbs MoE all-to-all wrapper. It bypasses the CPU proxy with GPU-driven raw verbs (WR-chaining, per-thread CQ/QP, out-of-order drain) and lands ~4× faster on the dispatch microbench — 47 µs vs DeepEP's 186 µs. It is the closest thing on EFA to what GDAKI would deliver, but implemented in user space.
Partial. It serves via stock SGLang
(--moe-a2a-backend deepep + a usercustomize shadow so every rank picks up
the UCCL wrapper). It did not serve stock vLLM — vLLM's per-DP-worker
Buffer construction deadlocks UCCL's collective all_gather_object; SGLang's
symmetric launch satisfies the contract.
Same framework (SGLang), same pods/model, ≥2-rep, decode variance <1%:
| regime | DeepEP-V2 tok/s | UCCL tok/s | DeepEP / UCCL |
|---|---|---|---|
| conc 32 | 263.9 | 254.2 | 1.038× |
| conc 256 | 2173.8 | 2105.4 | 1.032× |
measured UCCL's ~4× transport edge does not convert to serving throughput at EP16 — the bars on top are wildly unequal, the bars on the bottom are a tie. This is the same conclusion the whole campaign reaches from the other direction: all-to-all is not the serving critical path, so the fastest transport on EFA ties the proxy one when you actually serve.
When to pick it. As a research comparator, or where you can carry a fork and want the
robustness edge: DeepEP-V2's ElasticBuffer crashed on long-prompt prefill
(num_max_tokens_per_rank=128 cap) where UCCL completed every cell. For turnkey serving on
EFA today, UCCL's fork requirement and serving tie make it a harder sell than dense or DeepEP-multicomm.
Perplexity's open MoE all-to-all kernels (perplexity-garden). We built them on EFA and ran the transport benchmark to complete the comparison.
| path | EFA latency | vs published |
|---|---|---|
| dispatch | 222 µs | matches |
| combine | 362 µs | matches |
No — diagnostic comparator only. pplx has no serving adapter: there is no
vLLM / SGLang backend that drives it for real inference. It validates the
transport layer (and confirms our EFA build is sane — the numbers match published), but it is not a
turnkey serving backend.
Transport benchmarking and sanity-checking a build. Not for production serving on EFA without writing the missing serving integration yourself.
GPU-initiated communication: the kernel posts the work-queue entry and rings the doorbell itself — no CPU proxy on the data path. This is the only path that would let DeepEP beat dense on serving, because it removes the fixed +14.7 ms/token proxy tax that every measurement traces back to.
Use dense allgather (fastest on TRT-LLM, 1.5×) or DeepEP multi-comm / allgather where they tie (vLLM, SGLang). Pick on robustness and operational simplicity, since serving throughput is a wash.
If the model won't fit without sharding experts, use DeepEP / EP — a working serve beats OOM. That is an EP-itself win (allgather-EP has it too), not a DeepEP-a2a throughput win.
UCCL (fork, serving tie, long-prefill edge) and pplx (no serve adapter) are for benchmarking the transport layer — not turnkey serving backends on EFA.
DeepEP beats dense on serving only with GDAKI / IBGDA — which removes the proxy tax but is InfiniBand-only today. DeepEP's win on EFA is capacity + a future GDAKI.
Bottom line. On EFA today, for serving: dense (TRT) or DeepEP-multicomm / allgather-tie (vLLM, SGLang). UCCL / pplx = research comparators. DeepEP's real win is capacity + a future GDAKI path the shipping fabric stack cannot run yet. Update 2026-07-17: a native kernel-posted path (CQ-poll, no hardware counters) is now measured in §10 — it beats the proxy on dispatch/combine latency but remains a research path, not a serving recipe; this section's verdict is unchanged.