DeepEP on EFA
SECTION 06 / FINAL · solution architecture + deployment guidance

Solution architecture & guidance

The decision layer. Everything the prior sections measured, distilled into what to deploy and when — separated cleanly for two readers: the internal architect choosing a transport, and the customer who just needs a working MoE serve on AWS.

Executive summary. On AWS EFA, DeepEP MoE all-to-all is a working drop-in across three serving frameworks (SGLang, vLLM, TRT-LLM) — but it does not beat a dense / allgather alternative on serving throughput. The reason is structural: EFA's CPU-proxy transport path is issue-bound (~2.2% NIC utilization measured) and pays a fixed +14.7 ms/token measured per-decode-step tax that the all-to-all byte reduction cannot overcome. Use DeepEP on EFA for capacity (fitting a too-big model) or framework fit — not for a serving speedup. A serving speedup requires GPU-initiated transport, which no shipping EFA stack exposes today. Update 2026-07-17: a native kernel-posted path (CQ-poll, no hardware counters) is now measured in §10 — it beats the proxy on dispatch/combine latency but remains a research path, not a serving recipe; this section's verdict is unchanged. On InfiniBand the calculus flips: IBGDA posts the WQE from the kernel, removes the proxy tax, and DeepEP's transport win converts to a serving win.

Every measured claim below is tagged measured on the efa-gda campaign: 2× p5en.48xlarge, AWS EFA (16 NICs/node, 25 GB/s/NIC effective), Qwen3-30B-A3B-FP8 EP16 unless noted. derived = computed from measured. assumed = engineering judgement, verify locally.

1 · What works today on EFA — the proven stack

One stack serves DeepEP-V2 MoE all-to-all on EFA today. It is a binary drop-in: the deep_ep wheel and ElasticBuffer API are stock; the transport runs over the open-source aws-ofi-nccl GIN plugin via a host CPU proxy. Three frameworks ride it, all serve coherently, all pass AIPerf.

The data path measured

cross-node DeepEP-V2 dispatch on EFA (the working path) ┌──────────────┐ │ GPU kernel │ token routing, expert select (ElasticBuffer) └──────┬───────┘ │ writes descriptors ▼ ┌──────────────┐ │ D2H ring │ device → host signal ring └──────┬───────┘ │ wakes the host ▼ ┌──────────────┐ │ CPU proxy │ aws-ofi-nccl GIN, NCCL_GIN_TYPE=2 ◄── the +14.7 ms/token tax └──────┬───────┘ single ep_lock · issue-bound · 2.2% NIC util │ fi_writedata() ▼ ┌──────────────┐ │ libfabric │ efa-direct provider └──────┬───────┘ │ ▼ ┌──────────────┐ │ EFA SRD │ no atomics · no ordering · CQ-local · 1-SGE write └──────┬───────┘ │ ~25 GB/s/NIC × 16 ▼ ┌──────────────┐ │ peer GPU │ receiver polls (no recv CQE for unsolicited write) └──────────────┘

The pinned components measured

layerwhat / why
DeepEP APIdeep_ep.ElasticBuffer — the EFA-viable buffer (NCCL-GIN). Never deep_ep.Buffer.
GIN transportNCCL_GIN_TYPE=2 (CPU proxy). Type 3 = GDAKI = not shipping on EFA.
pluginaws-ofi-nccl 9c44d34 — combine-rewrite + FALLBACK_V1 gate-relax for gdrdrv-2.4 nodes.
parallelismEP_NUM_COMMS=8 — K independent NCCL comms = K proxy threads = 4.3× the issue-bound ceiling.
frameworksSGLang 0.5.11 · vLLM (deepep_v2) · TRT-LLM 0.21 — all serve, all pass AIPerf.

All three frameworks construct ElasticBuffer on EFA (SGLang/vLLM natively or via shim; TRT-LLM via the api-shim path). Coherent generation verified end-to-end on each.

⚠ DENYLIST — hard-banned on EFA. Do not construct deep_ep.Buffer (its cross-node transport is NVSHMEM/IBGDA, which aborts on EFA — nvshmemi_init_thread IBGDA path). Do not set NVSHMEM_IB_ENABLE_IBGDA=1. Use ElasticBuffer exclusively for cross-node MoE on EFA, and assert it in the harness — a silent fall-through to Buffer looks enabled but is wrong, and is the single most common deployment defect.

2 · Deployment requirements — the load-bearing config customer-actionable

These are not tuning knobs. Each one is a hard gate — get it wrong and DeepEP either fails to initialize or silently runs the wrong (forbidden) transport. Set all of them.

Pod must be privileged + mount host /dev

The GIN proxy opens /dev/gdrdrv for GPUDirect. Without a privileged pod and a hostPath /dev mount, gdr_open() returns EPERM → GIN logs "Failed to open gdr handle" → ginType==NONE → ElasticBuffer asserts at construction. measured

This is a manifest bug, not a plugin/gdrcopy-version bug — the most expensive one to misdiagnose. It is the load-bearing requirement.

EP16 mandatory for cross-node MoE

Expert-parallel all-to-all only engages when ep_size > top_k. For a top-8 MoE that means EP ≥ 16 (2 nodes × 8 GPUs). Below that, experts collapse onto fewer ranks and you are not exercising the cross-node path at all. measured

EP16 = the minimum honest 2-node configuration. Intra-node EP rides NVLink and pays no EFA tax.

Environment — the complete set

variablevaluewhy
NCCL_GIN_TYPE2CPU-proxy GIN. The only EFA-viable transport (3=GDAKI is upstream-gated).
EP_NUM_COMMS8Multi-comm. K proxy threads → 4.3× the single-comm 8 GB/s ceiling. measured
NCCL_NET_PLUGIN→ aws-ofi-nccl 9c44d34combine-rewrite + V1 gdr-pin fallback for gdrdrv-2.4 nodes.
SGLANG_DEEPEP_USE_V21SGLang: select DeepEP-V2 ElasticBuffer (not legacy Buffer).
SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK512Lifts the 128-tok/rank cap that crashes long-prompt prefill at conc≥32.
pod specprivileged: true+ hostPath mount of /dev — see above. Non-negotiable.

Framework selection of the backend differs (--moe-a2a-backend deepep on SGLang, --all2all-backend deepep_v2 on vLLM, AlltoallMethodType.DeepEP via shim on TRT-LLM) — but the transport stack below the framework is identical across all three.

3 · Internal solution-architecture decision tree internal

For the architect placing a MoE model on AWS. Follow the branch; the leaf is the transport choice. The pivot is always the same question — is the network InfiniBand or EFA, and is your goal throughput or capacity?

MoE model on AWS? choose the transport Fits in one node? (model + KV-cache vs 8× GPU mem) yes · 1 node Use NVLink EP all-to-all on NVLink · no EFA tax. DeepEP optional, no benefit vs dense. no · multi-node Which fabric? cross-node interconnect InfiniBand DeepEP + IBGDA WINS kernel-posted WQE removes the proxy tax; transport win → serving win. Use DeepEP. EFA Throughput or capacity? what is the binding goal throughput Measure your alternative first TRT dense: 1.5× faster → use dense. vLLM allgather: TIE → DeepEP-multicomm OK. DeepEP does not win throughput on EFA. capacity-forced Use DeepEP / EP to fit model too big for one node → shard experts. Accept the a2a toll; a working serve beats OOM. EFA throughput leaf: the alternative is a same-framework dense / allgather backend — pick the transport on measured tok/s, not vendor bytes. DeepEP's all-to-all is not the serving critical path on EFA. The fabric, not the kernel, decides the verdict.

Leaf colors: green = clear use, amber = measure / tie, purple = InfiniBand-only win. The EFA throughput branch is the one most likely to surprise — see the customer cards below for the per-framework call.

4 · Customer-facing guidance customer

Three intents, three answers. Pick the card that matches what you are actually trying to do.

✓

"Just want it to work"

DeepEP-V2 is a proven drop-in on EFA across all three frameworks. Deploy the stack from §2 verbatim: ElasticBuffer + NCCL_GIN_TYPE=2 + plugin 9c44d34 + EP_NUM_COMMS=8, privileged pod with /dev mounted.

All three serve coherently · all pass AIPerf · EP16 on 2× p5en. measured

▲

"Max throughput on EFA"

Measure your alternative. On TRT-LLM the dense path is ~1.5× faster — use dense. On vLLM it is a tie (allgather ≈ DeepEP-multicomm) — pick on ops. DeepEP does not give you a throughput edge here; choose on operations, not on the all-to-all.

TRT 0.67× · vLLM 0.97× · SGLang vs UCCL 1.03× (tie). measured

◆

"Model too big for one node"

Use EP / DeepEP for capacity — sharding experts is how a 235B / 256-expert MoE fits at all and frees KV-cache memory. Accept the all-to-all toll: the alternative is OOM, not a faster serve. This is an EP-itself win (allgather-EP has it too).

Capacity is the genuine, often-mandatory advantage — distinct from the contested throughput claim. verify against your GPU memory

The InfiniBand flip. Every "tie" or "use dense" verdict above is an EFA verdict. On an InfiniBand cluster, IBGDA posts the dispatch WQE directly from the GPU kernel, the +14.7 ms/token proxy round-trip disappears, and DeepEP's O(top-k) transport reduction converts into a real serving speedup. Same model, same framework — different fabric, opposite recommendation. assumed by architecture; EFA path is the measured one here

5 · Roadmap — what would make DeepEP win on EFA

The single lever that flips the verdict is GPU-initiated transport: post the dispatch WQE from the CUDA kernel itself (GDAKI), removing the host proxy round-trip. That converts DeepEP's measured transport win into a serving win. Here is exactly where that stands — honestly.

SHIPPING
today
CPU-proxy ElasticBuffer works

3-framework drop-in, EP16, multi-comm. Parity with dense/allgather; 1.45× dense on transport bytes but that never reaches tok/s. measured

GDAKI
barrier
Kernel-posted WQE — barrier path works

Cross-node DeepEP barrier runs GPU-initiated over EFA at ~19 µs. Proof the wire accepts kernel-posted traffic. measured

GDAKI
dispatch
Kernel-posted WQE — dispatch path blocked · 2 gates

Hard-blocked at runtime on a stock p5en (source-verified 2026-06): (1) NCCL's GDAKI backend is IB/RoCE-only by design — EFA needs a brand-new EFA-GDA backend that does not exist publicly yet (NCCL #2090, NVIDIA + AWS "actively working"). (2) The GPU-pollable completion counter it needs is unshipped: the EFA kernel module (efa_linux 3.1.0) has no comp_cntr object and efadv_query_device doesn't report the cap; rdma-core #1701 is open/no-ETA. libfabric #12114 + the plugin compile, but runtime dies at the efadv_create_comp_cntr() ioctl. EFA SRD also has no atomics/ordering, which the accumulating put-signal path assumes. Verdict: park; re-evaluate when an efa_linux release ships completion counters.

software
cheapest
Free env knob + upstream plugin rebuild untested · testable now

(0) VLLM_MOE_DP_CHUNK_SIZE sweep — the only runtime knob that packs more tokens per transfer (fewer messages = the right lever for an issue-bound fabric); free, no rebuild. (1) rebuild the plugin from aws-ofi-nccl@master (our shipped one is 64 GIN commits behind): commit a698b6e "move signal gdrcopy to a worker thread" takes the receiver read-modify-write off the proxy CQ-drain thread — directly attacks the per-step proxy stall, decode-concentrated. Builds on the CPU-proxy path (no GDAKI gate), ~15 min. untested; bounded by the 1-SGE cap

software
deeper
In-plugin K-thread rail driving scoped · GO

Shard the proxy iput post path across K dedicated post threads with per-shard request pools and a split ep_lock — attacks the issue-bound ceiling in software, no kernel change. Multi-day C++ build; the EP_NUM_COMMS=K user-space path already ships the throughput win so no serving deliverable is blocked. scoped, not yet built

Honest status: GPU-initiated dispatch on EFA is not shipping today. The nearer-term software path (K-thread rail driving) is the realistic next lever. Until one of these lands, DeepEP-on-EFA is a capacity / framework-fit tool, not a serving accelerator — exactly what §0 says.

When DeepEP on EFA is the right call
  • The model does not fit in one node (capacity-forced) — EP is mandatory.
  • Your framework's a2a backend is DeepEP and the alternative is heavier (vLLM allgather → tie).
  • You need the proven 3-framework drop-in and ops simplicity over a marginal throughput delta.
  • You are on InfiniBand, not EFA — then it wins outright.
When to pick the alternative
  • Throughput is the goal and the dense path is fast (TRT-LLM → dense ~1.5× faster).
  • You expected the all-to-all byte reduction to speed up decode — it doesn't on EFA's issue-bound proxy.
  • Expert count is very high (256) — the per-op tax anti-scales (0.36× vs 128 experts). measured
  • You constructed deep_ep.Buffer — that path is NVSHMEM/IBGDA and forbidden on EFA.

End of the DeepEP-on-EFA guidance site. ← Back to the Overview · run the numbers for your own model + workload in the DeepEP-on-EFA Calculator ▸

All measured claims sourced from the efa-gda campaign A/B experiments (TRT-LLM-vs-dense, DeepEP-multicomm-vs-dense, SGLang DeepEP-vs-UCCL, AIPerf 0.10.0). Sanitized — no IPs, account IDs, or hostnames.