DeepEP on EFA

§7 · The Math — why MoE all-to-all behaves differently on EFA vs InfiniBand measured-calibrated

A first-principles cost model for MoE expert-parallel all-to-all, derived from scratch and calibrated to measured numbers. The thesis in one line: DeepEP trades communication bytes for message count, and on a fabric whose bottleneck is per-operation issue rate (EFA's CPU proxy), increasing message count is exactly the wrong move. Below we build the model, plug in measured constants, and produce the closed-form serving-throughput gap between EFA and InfiniBand.

7.1 · Notation + measured constants

tokens/rank/step (decode, conc=32)
T ≈ 2
top-k experts/token
k = 8
EP degree
E = 16
cross-node fraction (2 nodes×8)
½
EFA per-NIC effective
25 GB/s
EFA node wire
400 GB/s
measured node injection ceiling
113 GB/s
measured dispatch NIC util
2.2%

All constants measured on 2× p5en.48xlarge EFA, Qwen3-30B-A3B-FP8 EP16 (per §4 microbenchmarks + MEASURED-DATA.md). The injection ceiling 113 GB/s (EXP-G40) proves the silicon is not the limit — the wire can do 3.4× what DeepEP extracts; the limit is above it, in software.

7.2 · How much does a MoE layer actually communicate?

Each MoE layer does two all-to-alls per forward pass: dispatch (scatter tokens to their experts' GPUs) and combine (gather results back). For T tokens/rank routed to k experts each, with experts sharded across E ranks:

# messages a rank must ISSUE per dispatch (one per distinct destination rank it has tokens for) msgs_dispatch = min(T·k, E−1) # bounded by #peers; at decode T·k is small → scatter to many peers bytes_dispatch = T·k·d_model·dtype # the PAYLOAD — this is what DeepEP minimizes (O(top-k), not O(experts)) # DeepEP's advantage is on BYTES: bytes(DeepEP) = T·k·hidden·1B(fp8) # only the k chosen experts' tokens move bytes(allgather) ≈ T·E·hidden·1B # every rank sees every token (E/k ≈ 2× more bytes at k=8,E=16)

So on bytes, DeepEP wins by ≈ E/k ≈ 2× at this config (more at larger E). That is real — and it is why DeepEP's transport microbench hits 33 GB/s SO vs UCCL-published 22.77 measured. But serving throughput is not set by bytes when the fabric is issue-bound. Next we show why.

7.3 · The two rooflines: byte-bound vs issue-bound

A transfer's time is the max of two limits — moving the bytes, and issuing the operations:

t_transfer = max( bytes/BW_wire , msgs/R_issue ) └─ byte-bound ─┘ └─ issue-bound ─┘ # BW_wire = 400 GB/s/node (or 113 measured injection ceiling) # R_issue = ops/s the proxy can POST. One mutex-serialized CPU proxy thread.
message size (bytes/op) — log scale → achieved GB/s 400 GB/s wire ceiling 113 GB/s measured injection ceiling issue-bound: BW = msgsize × R_issue DeepEP decode: ~128-tok msgs → 2.2% of wire DeepEP prefill: 4096-tok → 33 GB/s (8% wire) MoE all-to-all lives on the LEFT (small messages) → issue-bound, not byte-bound
The roofline. MoE dispatch messages are small (especially at decode), so the operating point sits on the steep issue-bound diagonal, far below the 400 GB/s wire ceiling — at a measured 2.2% utilization. Reducing bytes (DeepEP's lever) slides you slightly left along a line you're already under; it does not raise R_issue. measured

7.4 · The per-operation latency budget — where the time actually goes

The serving-throughput difference is set by the per-operation latency on the critical path, not bandwidth. Decompose one all-to-all op on each fabric:

InfiniBand + IBGDA (GPU-initiated, kernel posts the WQE): t = doorbell(~0.5µs) + RTT_ib(~2µs) ≈ ~2.5µs, fully GPU-issued, OVERLAPPABLE with compute EFA + CPU proxy (today — GPU cannot post; host must): t = D2H_ring_write + proxy_poll_latency + fi_writedata_build + libfabric + SRD_RTT + completion_poll = a HOST ROUND-TRIP, per op, NOT hidden behind compute measured aggregate effect on decode: +14.7 ms / token vs the dense/allgather baseline
per-decode-token latency budget (ms) dense/allgather compute + small a2a ≈ 36.3 ms → 27.6 tok/s DeepEP (EFA proxy) same compute 36.3 ms +14.7 ms proxy tax → 19.6 tok/s 0 36 51 ms The compute is identical. The ONLY difference is the red proxy round-trip — and it is on the critical path, not hidden.
Decode latency budget. Both arms do the same expert compute (~36.3 ms/token aggregate). DeepEP adds a fixed +14.7 ms/token CPU-proxy round-trip that IBGDA would hide. 1000/36.3 = 27.6 tok/s (dense); 1000/51.0 = 19.6 tok/s (DeepEP). derived from measured per-user rates.

7.5 · Why the gap is a CONSTANT ratio — the proof it's a fixed cost

If the proxy tax τ is a fixed per-step cost (independent of load), the throughput ratio is constant across workloads. Measured per-user tok/s is flat — DeepEP 19.6 (σ=0.13), dense 27.6 (σ=0.51) — across conc4→32 and ISL120→2048. That flatness is the proof:

ratio = tok/s(DeepEP) / tok/s(dense) = [1/(C+τ)] / [1/C] = C/(C+τ) with C = 36.3 ms (compute), τ = 14.7 ms (proxy tax): ratio = 36.3 / 51.0 = 0.71 # vs measured 0.65–0.72 across ALL regimes ✓ # a workload-dependent (bandwidth) effect would make this ratio MOVE with conc/ISL. It does not. # constant ratio ⇔ fixed per-step latency cost ⇔ NOT a bandwidth problem.
workload: conc4/ISL120 → conc32/ISL120 → conc16/ISL2048 per-user tok/s dense 28.1 / 26.9 / 27.8 DeepEP 19.7 / 19.4 / 19.7 constant gap = the fixed τ
Two flat lines. The gap between them does not narrow as load grows — that constancy (σ ≈ 0.1–0.5 tok/s) is the empirical signature of a fixed per-step cost, not a bandwidth effect a bigger batch could amortize. measured

7.6 · The closed-form EFA-vs-InfiniBand serving gap

On InfiniBand + IBGDA the all-to-all is GPU-issued and overlaps compute, so τ_ib ≈ 0 on the critical path. On EFA the proxy tax τ_efa is exposed. The serving-throughput gap between the two fabrics, for the same DeepEP algorithm, is:

speedup(IB / EFA) = tok/s_ib / tok/s_efa = (C + τ_efa) / (C + τ_ib) ≈ (C + τ_efa) / C = (36.3 + 14.7) / 36.3 = 1.40× # IB would serve DeepEP ~1.4× faster than EFA does # and vs the EFA dense baseline, DeepEP-on-EFA is 0.71× — i.e. on EFA the all-to-all OPTIMIZATION is a PESSIMIZATION, # because the fabric makes you pay for messages, and DeepEP issues MORE (smaller) of them than allgather.
The decisive asymmetry: InfiniBand rewards reducing bytes (it has GPU-initiated posting, so message overhead is cheap and bandwidth is the budget). EFA today rewards reducing message count (the host-proxy makes each op expensive, so issue rate is the budget). DeepEP optimizes the first and aggravates the second. Same code, opposite verdict — set entirely by whether the NIC can be driven from the GPU kernel.

7.7 · What would change the math (the improvement path)

The model says the lever is τ (the per-op proxy tax), not bytes. Two ways to attack τ:

1. GPU-initiated transport on EFA (GDAKI) — kernel posts the WQE, τ → ~one doorbell+RTT. Status: cross-node BARRIER works (19µs measured); DISPATCH blocked on rdma-core #1701 (the gap shrank from 3 unshipped layers to 1 — libfabric #12114 + aws-ofi-nccl GDAKI now upstream). Would collapse τ and FLIP the verdict — this is the InfiniBand-parity path. 2. Cut proxy ops per token (software, today) — eager-sends for grouped receives (aws-ofi-nccl #1194/#1211, merged, NEVER TESTED) + multi-comm K threads (EP_NUM_COMMS=8, ships 4.3× transport BW but doesn't yet convert to tok/s). These lower the constant in front of τ; they don't remove the host round-trip.

The honest ceiling: software levers (#2) can raise R_issue and shave τ but cannot make a host round-trip free. Only GPU-initiated posting (#1) removes it. Until rdma-core #1701 ships, EFA's MoE-serving verdict stays "dense/allgather ≥ DeepEP." measured + model.

·
All constants measured on 2× p5en.48xlarge AWS EFA, Qwen3-30B-A3B-FP8 EP16, AIPerf 0.10.0. Model derivations tagged derived/model; inputs measured. Sanitized — no internal IPs, registry IDs, or hostnames.