A primer for solution architects and systems engineers. We build up Mixture-of-Experts (MoE) from the dense feed-forward layer it replaces, isolate the two all-to-all communication phases that dominate the network, and explain why MoE serving is fundamentally a fabric problem. The thesis of this whole site lives here: DeepEP is engineered to shrink that all-to-all — and on AWS EFA the shrink does not turn into serving throughput. Section 2 shows the arithmetic.
Every transformer block has two halves: attention, then a feed-forward network (FFN). In a dense model, every token passes through the same FFN — every weight is touched on every token. A Mixture-of-Experts layer replaces that single FFN with N parallel FFNs called experts, plus a small router (a.k.a. gate) that scores the experts per token and sends each token to only its top-k of N. Total parameter count grows with N; the parameters that actually compute on a given token stay fixed at k experts. That decoupling of capacity from compute is the entire point of MoE.
Left: a dense FFN — one network, 100% of its weights multiply every token. Right: an MoE layer — the router scores all N experts and routes each token to its top-k (k = 2 drawn; green = active for this token). Capacity scales with N; compute scales with k.
The A3B suffix is read literally as "3 B active". This is the model we serve
end-to-end across all three frameworks in this site, so its numbers anchor every example.
| Total params | 30 B |
| Active / token | 3 B |
| Experts (N) | 128 |
| Top-k | 8 |
~10% of weights compute per token. 8 of 128 experts fire.
| Total params | ~47 B |
| Active / token | ~13 B |
| Experts (N) | 8 |
| Top-k | 2 |
Coarse-grained: 8 large experts, 2 fire per token.
| Routed experts | 160 / 256 |
| Top-k routed | 6 / 8 |
| Shared experts | 2 / 1 |
| Style | fine-grained |
Many small experts + always-on shared expert(s).
When experts are spread across multiple GPUs (expert parallelism, §3), one MoE layer is not one matmul — it is a four-phase pipeline, and two of those phases are collective network operations. This decomposition is the mental model for the rest of the site.
A tiny linear layer scores all N experts per token, takes the top-k, and produces a routing table: which token → which expert, plus per-expert combine weights. Microseconds of GPU math. No network.
Scatter. Every token is sent to the GPU(s) that hold its k chosen experts. Because routing is data-dependent, any token can target any expert on any GPU/node — an all-to-all. This is half of the MoE communication.
Each GPU runs its local experts over the tokens it just received — a batched grouped-GEMM. Heavy GPU math; no network, but it cannot start until dispatch delivers the tokens.
Gather. Expert outputs travel back to each token's origin GPU and are weighted-summed by the gate weights. A second all-to-all, the mirror of dispatch.
Phases (b) dispatch and (d) combine sit on the network lane; (a) router and (c) FFN are GPU compute. The two all-to-alls bracket the expert compute and cannot be hidden behind it — combine cannot start until the FFN finishes, which cannot start until dispatch delivers. This is the chain DeepEP tries to compress.
The routing table is computed fresh per token batch, so the traffic matrix is dense and irregular — every GPU talks to every other GPU. Combine is the transpose of this matrix. There is no static pattern a topology can pre-optimize away, which is why the fabric's latency and small-message behavior matter so much.
A 30B–671B-parameter MoE model does not fit, and should not fit, on one GPU. The natural way to split it is by the expert dimension: shard the N experts across EP GPUs. That is expert parallelism (EP). The instant experts live on different GPUs, a token whose top-k lands on a remote expert must travel there — and that is the all-to-all. EP is the parallelism strategy that creates the MoE network problem.
| Strategy | Splits | Collective |
|---|---|---|
| EP (expert) | experts across GPUs | all-to-all / layer |
| TP (tensor) | each weight matrix | all-reduce / layer |
| DP (data) | the batch | all-reduce / step |
| PP (pipeline) | layer ranges | p2p send/recv |
TP/DP use bandwidth-friendly all-reduce on large, regular tensors. EP uses all-to-all on small, irregular, per-token payloads — a far more latency-sensitive and small-message-sensitive collective. In production these compose (e.g. EP×TP×DP); EP is the term that puts the data-dependent all-to-all on the wire.
DeepEP (DeepSeek's expert-parallel communication library) is built specifically to make those two all-to-alls cheap. It is a genuinely strong design — on the fabric it was designed for. Three ideas do the work:
A naïve EP all-to-all can move O(N) token copies. DeepEP moves each token only to its k chosen experts — for Qwen3 that is 8 of 128, an order-of-magnitude cut in bytes on the wire versus a dense allgather-style exchange.
Routing layout, data movement, and the combine reduction are fused into dedicated GPU kernels — no slow host round-trips, no generic-collective overhead, tuned for the small, irregular MoE payload.
On InfiniBand, DeepEP posts RDMA work directly from the GPU via IBGDA (over NVSHMEM) — the CPU is off the data path entirely. This is what makes the latency-bound decode all-to-all fast.
On a fast, low-latency fabric — InfiniBand + NVLink + IBGDA — DeepEP's volume reduction and GPU-initiated, fused kernels combine into a large, real win for MoE serving. That is the regime DeepEP was authored in, and there it earns its reputation. The open question this site answers is what happens when the fabric underneath is AWS EFA instead.
Everything above changes character at one boundary: whether the all-to-all stays inside one server or must cross the network. Inside a node, GPUs talk over NVLink — terabytes per second, sub-microsecond, GPU-to-GPU. The moment EP exceeds the GPUs in one node, some fraction of every dispatch and combine must traverse the NIC to another node. That cross-node hop is where the fabric — EFA or InfiniBand — decides your serving throughput.
Left — EP ≤ one node (8 GPUs on a p5en): the entire dispatch/combine all-to-all rides NVLink's full-mesh, ~TB/s, sub-microsecond. Right — EP > 8: experts span nodes, so a portion of every all-to-all is forced through each node's NIC over the EFA (or InfiniBand) fabric. The cross-node leg is orders of magnitude slower and higher-latency than NVLink, so it dominates — this single boundary is the crux of the entire site.
AWS GPU instances (p5 / p5en) carry 8 GPUs per node. Production MoE serving routinely wants EP of 16, 32, or 64 — which is 2, 4, or 8 nodes. So cross-node all-to-all is not an edge case on AWS; it is the normal operating point. That makes the behavior of the cross-node fabric — and DeepEP's reliance on a GPU-initiated path that AWS EFA does not provide — the deciding factor for MoE serving throughput. Section 2 quantifies it.
Takeaways. (1) MoE decouples model capacity (N experts) from per-token compute (top-k), at the price of two data-dependent all-to-all collectives per layer. (2) Expert parallelism shards those experts across GPUs, which is what puts the all-to-all on the wire — and across nodes once EP exceeds 8 on AWS. (3) DeepEP is built to shrink that all-to-all via O(top-k) volume, fused kernels, and GPU-initiated IBGDA — a real win on InfiniBand. (4) On AWS EFA there is no IBGDA, so the question becomes whether the advantage survives. It does not convert to serving throughput — and §2 shows the arithmetic.
Next: §2 · EFA vs InfiniBand — the arithmetic of why →
DeepEP-on-EFA Guidance · §1 of 6 · for solution architects and technical customers