Mixture-of-Experts dispatch is a storm of small, latency-sensitive messages, not a bandwidth-bound bulk transfer. That single fact is why the same DeepEP code that flies on an InfiniBand / IBGDA cluster runs into a wall on AWS EFA. This section derives the wall from the wire up: the transport semantics, the per-operation latency math, the measured 2.2% NIC utilization, and the load-invariant +14.7 ms/token decode tax that is the headline result.
Throughout, color is consistent so you can read the diagrams at a glance: InfiniBand RC (ordered, atomic, GPU-initiated) versus AWS EFA SRD (scalable reliable datagram — no ordering, no atomics, host-proxied).
InfiniBand serving GPUDirect uses Reliable Connected (RC) queue pairs: a connection-oriented, in-order, fully-featured RDMA transport. AWS EFA uses Scalable Reliable Datagram (SRD): reliable delivery and multipath spraying for scale, but a deliberately thinner verb set. The differences below are not tuning knobs — they are properties of the wire, and each one removes a primitive DeepEP's fast path was built on.
| Property | InfiniBand · RC | AWS EFA · SRD | Why it matters for MoE dispatch |
|---|---|---|---|
| Message ordering | Ordered — RC delivers in send order | msg_order = [ ] — no ordering guarantee |
DeepEP's receiver advances a per-peer signal sequence in order; SRD multipath can reorder, so the plugin must serialize completions instead of pipelining them. |
| Hardware atomics | FI_ATOMIC — fetch-add, CAS in NIC | NO FI_ATOMIC — atomics absent from caps | The GPU-initiated signal path (DOCA put_signal) issues an RDMA atomic fetch-add to bump a
peer's arrival counter. On EFA that WQE is silently dropped — the signal never lands. |
| Write-with-immediate | WriteImm — data + 32-bit imm, one op | No WriteImm on the SRD RMA write | One-op "deliver payload + notify" is impossible; payload and notification become separate operations, roughly doubling the per-token op count. |
| Completion semantics | CQE ⇒ remotely visible (RC) | CQ is LOCAL — CQE = NIC accepted the WQE, not delivered | A sender-side completion proves nothing about the peer. Wire-level verification must read peer memory or HW counters — "N posts fired" is Layer-0 evidence only. |
| GPU-initiated comm | IBGDA — kernel posts WQE to NIC directly | GDAKI — present but not shipped end-to-end (3-layer stack gap) | Without a kernel-posted WQE path, every cross-node op detours through the host CPU proxy — the central cost this whole section quantifies. |
| Memory registration | rkey usable across the HCA | MR is per-NIC — an rkey from NIC 0 is invalid on NIC 1 | 16 NICs/node means 16 separate registrations and rkey tables; the plugin allgathers per-peer, per-NIC keys rather than one global key. |
| Scatter-gather on RDMA write | Multi-SGE gather supported | RDMA write is 1 local SGE (send-only inline differs) | You cannot gather "payload + co-located count" in one WQE — the trick that would halve dispatch ops is
rejected at the descriptor level (EFA_IO_TX_DESC_NUM_RDMA_BUFS = 1). |
| Multipath | Single path per QP | Spray across paths — scales to many flows | EFA's strength: it scales to thousands of flows without HoL blocking. The cost is the reordering above — a trade made for scale-out, not bandwidth per flow. |
MoE dispatch sends each token's hidden vector to its top-k experts, which are scattered across the EP group. At decode the message is tiny — one token's worth, a few KB — and there are many of them. So the cost that matters is per-operation issue latency × operation count, not aggregate bandwidth. Here the two fabrics diverge sharply.
The GPU kernel builds the work-queue entry in device memory and rings the NIC doorbell itself. The host CPU is never on the data path.
No kernel-posted WQE path is shipped. The kernel hands off to a CPU proxy, which calls libfabric, which posts to the NIC. Every op is a GPU→host→NIC round-trip.
Two structural facts make the EFA path worse than the raw round-trip suggests, and both come straight from §1:
t_op is itself larger
derived.Decode is the worst case: a single token per user per step, dispatched to top-k experts, every
forward pass. Bandwidth is irrelevant — you are paying t_op many times in series, and on EFA each
t_op carries a host round-trip that InfiniBand's IBGDA does not.
If the wire were the bottleneck, the EFA NICs would be saturated during dispatch. They are not. On the DeepEP dispatch path the measured NIC utilization is ~2.2% — the fabric is 97.8% idle. The limit is the rate at which the host proxy can issue operations, not the rate at which the wire can carry bytes.
Issue-bound, not wire-bound. DeepEP's whole value proposition is reducing all-to-all
bytes (it sends each token to only its top-k experts, an O(top-k) volume instead of
all-experts). But when the wire is 97.8% idle, sending fewer bytes does not make dispatch faster — you are
gated on per-op post rate through the host, and that rate is unchanged whether the payload is large or small.
measured This is the crux: byte-reduction is the wrong lever for an issue-bound
path.
This is the headline result, and the proof is in the shape of the data. Across a wide sweep of concurrency (4 → 32) and input length (ISL 120 → 2048), the DeepEP per-user decode throughput does not move: it sits at 19.6 tok/s (σ = 0.13). The same-framework dense / allgather arm is also flat, at 27.6 tok/s. Two flat lines, separated by a constant gap.
Constancy is the evidence. If the gap were queueing or contention, it would grow with
load. It does not. A throughput that is flat in tok/s across a 8× concurrency range and a 17× ISL range can only
be explained by a fixed per-decode-step cost — the CPU-proxy host round-trip, paid once per token step
regardless of how much work surrounds it. The arithmetic:
1000/19.6 − 1000/27.6 = 51.0 − 36.3 = +14.7 ms/token derived.
Measured DeepEP/dense ratios: 0.65× (conc4) · 0.69× (conc32) · 0.67× (conc16/ISL2048)
measured — three regimes, same tax.
There is exactly one thing that removes the tax: a kernel-posted WQE on EFA — the EFA
analogue of InfiniBand's IBGDA, called GDAKI (GPU-Direct Async, kernel-initiated). If the GPU could ring
the NIC doorbell itself, the host round-trip disappears and t_op collapses toward the IBGDA figure
in §2. The honest status: part of it works, the part you need does not.
efa-direct provider, per-peer/per-NIC rkey allgather.put_signal fetch-add is
dropped) and no ordering (the in-order receiver stalls). Today, dispatch must use the
CPU proxy (NCCL_GIN_TYPE=2).And even the doorbell path that the barrier uses is not a free upgrade: the hardware-counter completion path GDAKI dispatch would need depends on a three-layer stack that is not shipped end-to-end:
Combined with SRD's no-ordering / no-atomics, this is why the proxy is mandatory on EFA today for dispatch. GDAKI is the right answer architecturally — it is simply not a shipping option on this fabric yet. Anyone telling you to "just turn on GPU-initiated dispatch on EFA" is describing InfiniBand. Update 2026-07-17: our §10 campaign measured a counter-free native variant (CQ-poll completion) that runs dispatch+combine correctly and beats the proxy — see §10; the shipping-stack statement above still holds.
Putting it together: the EFA path traverses the full host stack on every all-to-all op; the IBGDA path skips the host entirely. The latency lives in the layers the orange path must climb and the violet path does not.
EFA (orange): every op climbs five layers — kernel → D2H ring → CPU proxy → libfabric → NIC — before reaching the wire, and the wire sits 97.8% idle waiting for the next issue. IBGDA (violet): the kernel posts the WQE and rings the doorbell directly; the entire host segment is skipped. The latency this whole section quantifies is exactly the height of the orange stack that the violet path does not climb. The D2H ring, CPU proxy, and libfabric layers are measured on the live path; the IBGDA per-op figure is derived from the §2 model.
docs/invariants.md, deepep-calculator/MEASURED-DATA.md.
Provenance tags — measured on hardware ·
derived by arithmetic from measured values ·
assumed pending verification.