Mixture of Experts (MoE)
Master Mixture of Experts: sparse model architectures, expert routing, scaling strategies, and efficient large model inference.
What is a Mixture of Experts?
A Mixture of Experts (MoE) layer replaces one dense neural-network block with many expert blocks and a learned router. For each token, the router selects only a small number of experts, those experts process the token, and their outputs are combined before the model continues.
MoE lets a model store much more expert capacity without running every expert for every token. That makes sparse scaling attractive, but it does not make the system cheap by default: all expert weights must be stored, tokens may cross devices, popular experts can overflow, and a failed expert can interrupt the token path.
The invariant to remember
Every admitted token must reach the intended number of available experts within capacity, then return one correctly weighted result. Sparse compute is useful only while routing quality, load balance, communication, and degraded behavior preserve that invariant.
Review model serving first if batching, tail latency, and distributed inference are unfamiliar.
Follow one token through the sparse layer
The router is a small learned network, not an external load balancer. It makes a model decision for every token and produces both expert choices and combination weights.
1 Router
Score experts
Project the token representation into one score per expert, then normalize or otherwise rank those scores.
2 Admission
Select top-K
Choose the allowed expert destinations and check each destination against its per-batch capacity.
3 Communication
Dispatch tokens
Group token representations by expert and move them to the devices that own the selected weights.
4 Sparse compute
Run experts
Execute only the selected feed-forward networks while recording overflow, queueing, and device failures.
5 Return path
Combine outputs
Move expert results back to token order and apply the router weights before the residual connection continues.
Top-1 routing executes one expert per token. Top-2 can increase expert diversity and model quality, but it roughly doubles expert assignments and dispatch volume for the same token batch.
Size expert capacity before choosing hardware
For T tokens, K selected experts per token, E total experts, and capacity factor C, a useful first estimate is:
- Total assignments:
T x K - Balanced assignments per expert:
(T x K) / E - Capacity per expert:
ceil(C x T x K / E) - Overflow: assignments above an expert's capacity must drop, reroute, wait, or use a fallback
The average is not enough. Production planning needs a measured hot-expert multiplier from real token slices because domain shifts, languages, modalities, and prompt templates can change routing concentration.
Can this batch fit without dropping routed tokens?
Size per-expert capacity and expose the memory and network costs that sparse activation does not remove.
2,560
2,048 assignments at perfect balance
21.9%
Assignments that must drop, reroute, or wait
1.00 GiB
20.0 ms serialization floor at line rate
8 GiB
6.3% of experts execute per token
Illustrative expert load
The marker is each expert's capacity. Red bars cross it.
E1
E2
E3
E4
E5
E6
E7
E8
The hottest expert drops or reroutes work
Capacity is below the predicted hot-expert load. Raising capacity may contain the symptom, but the durable fix is better routing balance or replicated hot expertise.
Increasing the capacity factor treats overflow by reserving more expert slots. It does not correct a collapsed router, add network bandwidth, or make failed expert weights available.
Separate active compute from resident memory
Sparse activation changes which expert parameters execute for a token. It does not remove the other experts from the checkpoint or accelerator fleet.
K / E
Active expert fraction
Top-K selects K of E expert blocks for each token; shared attention and other dense blocks still execute.
E copies
Resident expert weights
Every unique expert must be stored somewhere that the serving topology can reach.
T x K x H
Dispatch payload
Token assignments move hidden vectors of width H across the expert-parallel group and back.
p99
Operational target
The slowest selected expert or path can hold up the token, so tails matter more than averages.
Include the costs sparse FLOPs hide
- Expert checkpoints, optimizer states during training, replicas, and rollout overlap
- Dispatch and combine buffers, collective-operation workspace, and activation memory
- Shared attention, embeddings, normalization, router compute, and key-value cache
- Token padding, capacity reserve, dropped-token handling, retries, and fallback execution
- Cross-host serialization, queueing, synchronization, and straggler amplification
Choose a routing contract deliberately
Top-K routing
Token choice
Each token selects its highest-scoring experts. The model has flexible specialization, while capacity and balancing controls must contain hot experts.
Top-1
Switch routing
Each token selects one expert. Dispatch is smaller and simpler, but one bad choice or unavailable expert has no second selected path.
Bounded intake
Expert choice
Each expert selects tokens up to a fixed capacity. Expert load is bounded by construction, while token coverage and combination semantics need explicit validation.
Hash or rules
Deterministic routing
Assignments are reproducible and operationally simple, but the router cannot learn semantic specialization from the model objective.
Define overflow semantics as part of correctness
- Drop: fast and bounded, but the residual path receives no expert contribution for that assignment.
- Reroute: preserves work when a compatible expert has room, but changes model semantics and adds latency.
- Queue: preserves the selected expert, but lets one hot expert determine batch tail latency.
- Fallback: uses a dense or shared expert, but must be trained and evaluated as a real degraded mode.
Build the distributed path around expert ownership
An MoE layer is a distributed join
The dispatcher must restore every admitted token to its original position with the correct expert weights and failure evidence.
Score and admit
Router
Produces expert IDs, combination weights, capacity decisions, and routing telemetry for every token.
Regroup vectors
All-to-all dispatch
Packs token representations by destination and moves them across the expert-parallel topology.
Execute sparse FFNs
Expert owners
Run local experts under bounded capacity while exposing queue, overflow, device, and version state.
Restore token order
Combine
Returns expert outputs, applies router weights, validates coverage, and reconnects the residual path.
Compose parallelism instead of treating it as one setting
- Data parallelism replicates a model partition for independent batches.
- Tensor parallelism splits dense matrix operations across devices.
- Pipeline parallelism assigns layer ranges to stages and schedules microbatches.
- Expert parallelism distributes expert weights and creates the token all-to-all path.
- Sequence or context parallelism partitions long-sequence work and changes activation movement.
The placement plan must name device groups, fault domains, collective topology, checkpoint ownership, and which dimension can scale independently.
Diagnose the failed stage before changing capacity
The same user-visible symptom can come from different stages. A slow MoE layer may have a collapsed router, a congested all-to-all path, a straggling expert, a failed device, or expensive output synchronization. One universal tuning change usually moves the bottleneck instead of removing it.
Diagnose the failed stage before changing capacity
Loading incident evidence...
Train the router without erasing specialization
The primary model loss teaches task quality. Router-specific objectives shape how that quality is distributed across experts.
Distribute assignments
Load-balancing loss
Penalizes traffic concentration using router probabilities, realized assignments, or both. Too little permits collapse; too much can force unrelated tokens into unsuitable experts.
Stabilize logits
Router z-loss
Discourages router logits from growing to extreme magnitudes, which can improve numerical stability and preserve useful gradients.
Bound work
Capacity and admission
Caps per-expert assignments for a batch. This is a systems constraint whose drop or reroute behavior must also be part of model evaluation.
Explore carefully
Routing noise
Can prevent early collapse and expose experts to more tokens during training. Inference behavior must remain deterministic enough to operate and reproduce.
Evaluate the router as a model component
- Task quality by expert, token slice, language, domain, sequence position, and routing pattern
- Assignment entropy, expert load distribution, overflow, reroute, and drop rates
- Stability across seeds, checkpoints, precision modes, batch shapes, and distribution shifts
- Expert similarity, specialization, dead experts, and sensitivity to disabling one expert
- Calibration of router probabilities when they are used as combination weights
Operate MoE with layer-level evidence
End-to-end latency alone cannot identify whether routing, communication, expert compute, or combination failed.
Decision
Routing health
Track entropy, top-K margins, assignment counts, imbalance, overflow, drops, reroutes, and auxiliary losses by layer and token slice.
Movement
Fabric health
Track bytes, queue time, collective duration, source-destination skew, link saturation, and timeout rate for dispatch and combine.
Execution
Expert health
Track queue depth, batch size, compute time, memory, errors, weight version, replica state, and stragglers per expert owner.
Gate releases on failure behavior
- Replay representative routing traces against the candidate router and placement plan.
- Compare quality, overflow, all-to-all volume, and tails across production token slices.
- Remove one expert owner and verify rerouting, replica, or fallback semantics under load.
- Canary router and expert-weight versions together; reject mixed versions unless compatibility is explicit.
- Roll back on quality slices, dropped-token rate, expert imbalance, dispatch p99, or degraded-mode failure.
Know when dense is the better architecture
MoE earns its complexity when extra resident parameters improve quality enough to justify distributed routing and operational controls.
Sparse scale pays
Prefer MoE
The workload benefits from additional expert capacity, large batches amortize collectives, the fleet has fast topology-aware interconnects, and the team can operate routing evidence.
Simplicity pays
Prefer dense
Small batches, strict tails, limited memory, weak interconnects, edge deployment, or modest quality gains make predictable dense execution more valuable.
Reduce blast radius
Pilot a hybrid
Use MoE in selected feed-forward layers, keep shared experts or dense fallback paths, and validate one topology before scaling expert count.
Measure the frontier
Reject parameter theater
Compare quality, active FLOPs, resident memory, p99 latency, energy, availability, and operating burden against a tuned dense baseline.
The production question is not how many parameters the checkpoint contains. It is whether sparse capacity improves the measured quality-cost frontier while every token still completes a bounded, observable path.