Multimodal MoE Systems
Design multimodal MoE systems with sparse routing, text-image fusion, expert capacity, serving trade-offs, and evals.
What is a multimodal mixture-of-experts system?
A multimodal mixture-of-experts (MoE) system processes more than one input type, such as text and images, while activating only a small part of a much larger model for each token. A learned router scores a pool of expert networks and sends each token to one or a few experts.
In plain language: text and visual tokens can share one model, but they do not all take the same compute path.
The core invariant is every admitted token must have a bounded, observable path through the sparse layer. Sparse activation saves compute only when routing, capacity, communication, and overflow behavior remain controlled. It does not mean the inactive expert weights disappear from memory.
N experts
Stored capacity
All expert weights must be placed or made available to the serving system
Top-k
Active path
Each token activates only the selected routed experts, plus any shared expert
C
Per-expert capacity
A bounded number of token assignments may enter each expert for one routing group
Slices
Quality evidence
Grounding and task quality must be measured by modality, route, and overload state
Trace the architecture from media to sparse compute
Multimodal MoE is not one fixed blueprint. A system may use separate modality encoders before fusion, or it may place text and image tokens into one early-fusion backbone. The sparse part commonly replaces selected feed-forward sublayers inside a Transformer; attention and other layers can remain dense.
One multimodal sparse layer
The router acts on token representations, dispatches bounded token assignments to expert devices, then combines weighted expert outputs back into the original sequence.
Represent
Encode media
Tokenize text and convert images, regions, audio, or video into model-compatible token representations.
Align
Fuse the sequence
Place modalities in a shared representation space while preserving position, modality, and evidence boundaries.
Route
Score and select
Produce expert scores per token, select top-k destinations, and apply gates or balancing bias.
Communicate
Dispatch with limits
Group token assignments by destination, enforce expert capacity, and exchange them across expert-parallel devices.
Restore order
Combine and continue
Weight expert outputs, restore sequence order, add the residual path, and continue through later layers.
Keep four scopes separate
- Tokenization scope: decides how much text, image, audio, or video evidence enters the sequence.
- Routing scope: decides which expert paths process each token at a particular sparse layer.
- Capacity scope: decides how many assignments each expert can accept in one routing group or batch.
- Serving scope: places experts, batches requests, performs dispatch and combine communication, and enforces latency policy.
A request-level model router that chooses one whole model is not the same mechanism as a token-level MoE router inside a model.
Choose a routing contract, not just an expert count
One sparse destination
Top-1 routed
Each token takes one selected expert path. Active compute is predictable, but a hot expert has no second expert path unless the implementation defines one.
Several destinations
Top-k routed
Each token activates several experts and combines their outputs. This can add representation capacity while increasing dispatches, communication, and expert work.
Common and specialized
Shared plus routed
A shared expert processes every token while routed experts add conditional capacity. The shared path can capture common behavior, but it is real compute, not a free fallback.
Meta's Llama 4 model card is a current example of early-fusion multimodality with alternating dense and MoE layers. Maverick sends each token to a shared expert and one routed expert. That is one concrete architecture, not a rule that all multimodal MoEs follow.
The router needs an explicit contract
- State whether selection is top-1, top-k, shared-plus-routed, expert-choice, or another measured policy.
- Define the routing group over which expert capacity is calculated.
- Record whether gates are renormalized after selection or fallback.
- Preserve modality and task labels in traces without feeding private raw content into telemetry.
- Version the router, expert set, placement plan, capacity factor, and overflow policy together.
Route modalities without hiding token competition
Text and image tokens can share a router, yet their counts and representations differ. A high-resolution image may contribute far more tokens than its short question. Global balance can therefore look healthy while one modality receives poor routes, or one modality can dominate the gradients that shape the router.
LIMoE demonstrated that a shared sparse expert pool can learn image-text representations and that balancing must account for multimodal training behavior. Its entropy-based regularization encourages useful expert participation without requiring experts to be manually labeled as “text” or “vision.” Use the lab to compare routing contracts; the visible expert names describe observed roles, not hard-coded guarantees.
Balance specialization against collapse
A useful router must preserve two properties at once:
- Specialization: similar tokens can repeatedly find experts that improve the task loss.
- Utilization: the router cannot send nearly everything to a small set of experts while the rest stop learning or sit idle.
Training systems often add an auxiliary load-balancing objective, a router z-loss, balancing bias, capacity controls, or a combination. The exact mechanism is model-specific. A perfectly uniform token count is not automatically ideal because modalities and token types are not interchangeable.
Inspect more than average expert load
- assignment count and gate mass by expert, layer, modality, task, and sequence position;
- entropy or concentration of expert use, including underrepresented modalities;
- routing churn when the router or expert checkpoint changes;
- task loss and grounding quality under counterfactual or ablated routes;
- expert-to-device communication volume and the slowest device in each dispatch group.
Calculate capacity before deciding overflow behavior
For a token-choice router, a useful planning estimate is:
capacity per expert = ceil(tokens in routing group x top-k x capacity factor / routed experts)
The formula counts assignments, not original requests. Shared experts and implementation-specific rounding need separate accounting. A capacity factor above 1.0 provides headroom, but it can reserve more memory or compute and cannot correct a systematically bad route.
1 Router
Score each token
Compute expert logits from the token representation and apply the model's routing rule.
2 Top-k
Select destinations
Turn scores into bounded assignments and preserve the gate values needed to combine outputs.
3 Per expert
Enforce capacity
Accept assignments up to the expert limit, then classify every overflowed assignment explicitly.
4 Audit
Conserve the batch
Verify that direct, fallback, queued, and dropped assignments reconcile to total routed assignments.
This runnable model calculates routed load and reports the busiest expert instead of assuming that top-k creates balanced work.
Treat token overflow as a model behavior
When an expert is full or unavailable, the system still needs a defined result. “Drop” in sparse layers usually means skipping that expert transform for an assignment while the residual path continues; it is not necessarily an HTTP error and can silently change answer quality.
Drop assignment
Skip the sparse transform
Preserves bounded expert work and latency, but removes conditional compute from the affected token. Measure which modalities and evidence tokens are dropped.
Second choice
Try another expert
Can preserve sparse processing when spare capacity exists, but changes expert semantics, gate normalization, communication, and load on the fallback destination.
Fallback expert
Use a shared path
Provides a known general path when the architecture supports it. It raises active compute and may become a new shared bottleneck during broad failures.
Admission rejection, request queueing, and token overflow are different control points. Rejecting before inference protects a service SLO; skipping an internal expert assignment changes model execution after work has already begun.
Inject pressure before choosing a fallback
The same overflow policy behaves differently during a vision-token burst, a failed expert, and a balanced batch. Change the capacity factor and recovery policy, then trace where each overflowed assignment goes. The model conserves assignments so a low cost cannot hide dropped evidence.
Simulate overflow as a conservation problem
The example below separates direct processing, rerouted processing, shared fallback, and dropped assignments. It also reports a weighted evidence-loss score because dropping a layout token and dropping a low-salience background patch need not have the same product impact.
Place experts around the communication path
Expert parallelism shards experts across accelerator devices. A sparse layer therefore includes dispatch and combine communication, often an all-to-all exchange, in addition to expert computation.
Plan the physical system
- Co-locate or replicate hot experts only with measured activation traces and memory constraints.
- Batch enough assignments per destination to use efficient kernels without violating token latency.
- Track bytes sent, per-peer skew, stragglers, and dispatch/combine time by layer.
- Keep expert placement compatible with checkpoint loading, failover, and rollback time.
- Separate prefill and decode traces because their token shapes and latency sensitivity differ.
Fewer active parameters can reduce arithmetic work while total expert weights, routing exchanges, and device imbalance still dominate memory or tail latency. Report active FLOPs, resident bytes, network bytes, and goodput separately.
Evaluate the route as part of the model
Task
Answer quality
Compare dense and sparse baselines on the product's actual multimodal tasks
Ground
Evidence use
Test whether outputs follow the relevant image, region, document, or audio evidence
Route
Expert health
Measure concentration, modality coverage, overflow, fallback, and routing churn
SLO
System result
Measure goodput, tail latency, communication, memory, and cost under realistic traffic
Build evaluation slices that can reveal routing harm
- text-only, image-only, and interleaved text-image requests;
- image count, resolution, patch count, text length, and total routed assignments;
- OCR, document layout, spatial reasoning, fine-grained recognition, and conversational tasks;
- expert ID, sparse layer, gate confidence, capacity state, overflow policy, and failed device;
- safety category, language, accessibility use case, and user-impact severity.
Compare normal operation with capacity pressure. A model that passes only when no expert is full has not established a production quality contract.
Roll out routing changes with bounded exposure
1 Offline
Replay recorded token shapes
Use privacy-safe traces to reproduce modality mix, sequence length, expert demand, and communication skew.
2 No user effect
Shadow the new route
Compare assignments, gates, overflow, latency, and quality evidence without serving the candidate output.
3 Bound exposure
Canary by route version
Pin model, router, experts, capacity, placement, and overflow policy so rollback restores one coherent contract.
4 Quality and SLO
Promote on joint evidence
Require passing modality slices, capacity drills, communication targets, and cost-per-useful-answer thresholds.
Do not optimize the router in isolation. A lower auxiliary routing loss does not prove better multimodal answers, and a balanced training batch does not prove balanced production traffic. Ship the router, expert checkpoint, placement, capacity, overflow policy, and evaluation evidence as one versioned system.
Primary sources
- Switch Transformers defines a simple top-1 sparse Transformer, expert capacity, token dropping behavior, and load-balancing methods.
- V-MoE applies sparse experts to vision tokens and demonstrates adaptive per-image compute.
- LIMoE studies one sparse expert pool for language and image learning, including multimodal utilization and entropy regularization.
- DeepSeekMoE motivates finer routed experts and shared experts as distinct capacity paths.
- Llama 4 Maverick model card documents a current early-fusion multimodal architecture with a shared expert and routed experts.