A model with a trillion parameters sounds like it should cost a trillion parameters' worth of compute to run. For years that was roughly true — every parameter in a dense neural network gets multiplied against every input, so bigger models meant proportionally bigger bills, both to train and to serve. Mixture of Experts breaks that link. It's the architectural trick that let AI labs keep scaling model size without scaling inference cost at the same rate, and it's quietly underneath a large share of the frontier models released since 2023.
If you've seen a model card mention "8x7B," "236B total / 21B active," or "sparse MoE," this is what that notation is describing. Here's what the architecture actually does, why it became the default choice for large models, and where it still falls short.
The idea itself isn't new — conditional computation and expert-based networks show up in machine learning research going back decades, long before transformers existed. What changed is that the transformer architecture, combined with modern accelerator hardware, made it practical to apply the idea at the scale of hundreds of billions of parameters, and the economics of running models in production made the payoff worth the added engineering complexity.
What Mixture of Experts Actually Is
A standard ("dense") transformer has one feed-forward network per layer, and every token that passes through the model activates all of it. If the feed-forward block has 10 billion parameters, every single token — whether it's a comma, a proper noun, or a line of Python — pushes through all 10 billion.
Mixture of Experts replaces that single feed-forward block with several smaller ones, called experts, plus a small routing network that decides which experts should handle each token. Instead of one 10-billion-parameter block, you might have eight 10-billion-parameter experts — 80 billion parameters in total — but a router that sends each token to only two of them. The token gets processed by 20 billion parameters' worth of computation, not 80 billion, even though the model has eight times more capacity sitting on disk.
That's the core trade the architecture makes:
- Total parameters grow, which increases the model's capacity to store knowledge and specialize.
- Active parameters per token stay small, which keeps the actual compute cost of a forward pass low.
This is why MoE models are usually described with two numbers, like "141B total, 39B active." The first number tells you how much memory and storage the model needs. The second tells you roughly how much compute — and therefore how much latency and cost — each token actually consumes.
How the Routing Actually Works
The mechanism doing the real work is the gating network, a small trainable layer that sits in front of the bank of experts at each MoE layer.
The gating network
For every token, the gate produces a score for each expert — essentially a prediction of how useful that expert would be for this particular token. It then picks the top-k highest-scoring experts (commonly k=1 or k=2) and routes the token to only those. The outputs of the selected experts are combined, usually weighted by the gate's own confidence scores, and passed on to the next layer. This trainable top-k gating approach was formalized in the original sparsely-gated mixture-of-experts paper, which showed it scaling to thousands of experts on production hardware.
Crucially, the gate is learned end-to-end along with everything else. Nobody hand-assigns "this expert handles legal text" or "that expert handles code." Specialization — to the extent it happens — emerges from training, and in practice it's often messier and less interpretable than the "one expert per topic" mental model suggests. Studies of trained MoE models have found experts that seem to specialize by surface features (punctuation, token position, syntax patterns) as often as by subject matter.
Sparse activation
The reason this saves compute is that the unselected experts do nothing at all for that token — no multiplication, no memory read, no cost. This is called sparse activation, as opposed to the dense activation of a standard transformer where every parameter touches every token. Sparsity is the entire economic argument for MoE: you're buying a much larger, more expressive model but only paying the compute bill for a small slice of it on any given step.
Load balancing
Left alone, a trained gate will often converge on using only a handful of favorite experts, ignoring the rest — a failure mode called routing collapse. If that happens, you've paid to store dozens of experts but only a few ever get trained properly or used at inference time, which defeats the purpose.
To prevent this, MoE training adds an auxiliary load-balancing loss that penalizes the model for routing tokens too unevenly across experts — an approach Google's Switch Transformer research helped popularize at trillion-parameter scale. This pushes the gate toward spreading tokens more uniformly, which keeps all experts trained and useful, but it's also an extra term the model has to trade off against actually getting the right answer — an added source of training instability that dense models don't have to deal with.
Why This Matters Right Now
The pressure driving MoE adoption is straightforward: model quality has kept scaling with parameter count, but inference happens far more often than training, and inference cost is what shows up on a company's bill every single day a product is live. A model that's twice as good but ten times more expensive to serve is a hard sell for anything running at real usage volume — chat products, coding assistants, search, customer support.
MoE offers a way to chase the quality gains from scale — more parameters, more specialized capacity, better performance on rare or technical inputs — while keeping the per-query compute cost close to that of a much smaller dense model. That's why it has become a common architectural choice across multiple AI labs building large general-purpose models, rather than a niche research technique. It doesn't eliminate the cost of scale — training and hosting a model with hundreds of billions of total parameters is still expensive in memory and engineering effort — but it decouples "how smart the model can be" from "how much compute every single response costs" more than a dense architecture does.
Benefits of Mixture of Experts
The architecture's advantages all follow from separating total capacity from per-token compute. Here is what that separation buys model builders and the teams that use their models.
More capacity for the same inference compute
An MoE model can hold several times more parameters than a dense model with the same per-token cost. Those extra parameters give the model more room to store facts, patterns and rare skills. For a lab, that means chasing the quality gains associated with scale without multiplying the compute every response consumes. For users, it often means a stronger model at a price closer to that of a smaller one.
Lower serving cost at high volume
Inference happens far more often than training, so per-token compute dominates the lifetime cost of a popular model. Activating only a fraction of parameters per token reduces the arithmetic each request requires. At the scale of chat products, coding assistants and APIs serving millions of requests, that reduction is the difference between a model that is economical to offer and one that is not.
Faster training for a given quality target
Because each token only updates a few experts, an MoE model can reach a target quality with less training compute than a dense model of comparable capacity, a result reported in the Switch Transformer work and later research. That makes larger experiments affordable and shortens iteration cycles for labs, although the added training instability offsets some of the gain.
Room for specialisation
Different experts can learn to handle different kinds of input. The specialisation is often messier than "one expert per topic", but having multiple parameter sets lets the model represent diverse patterns, such as code, multiple languages and technical notation, without every token paying for all of them.
Flexible scaling
Capacity can be increased by adding experts rather than making every layer wider, and serving systems can spread experts across devices. That gives model designers another axis to scale along, alongside depth, width and data, and lets them tune the balance between memory and compute to the hardware available in a given data centre.
Mixture of Experts Use Cases
MoE is mostly something model builders choose, but it shows up in the products teams use every day. These are the settings where the architecture is most common.
Frontier general-purpose models
Problem: Large chat and reasoning models need enormous capacity, but serving them densely would be prohibitively expensive at consumer scale. How it's applied: Labs build models with many experts per layer and a router that activates a few per token, advertised with total and active parameter counts. Outcome: Large-capacity models served at a cost closer to that of much smaller ones, which is why MoE has become a common choice among large model releases since 2023.
Open-weight models for self-hosting
Problem: Organisations want capable models they can run on their own infrastructure for privacy, control or cost reasons. How it's applied: Several open-weight releases use MoE, described with notation like "8x7B", giving strong quality for their active compute. Outcome: Good performance per unit of compute, with the caveat covered below that every expert still needs to fit in GPU memory.
High-volume API and product serving
Problem: Coding assistants, search and customer support products process vast numbers of tokens every day, and cost per token decides margins. How it's applied: Providers serve sparse models with batching and expert placement tuned to their hardware. Outcome: Lower cost per request at volume, passed on partly as lower API prices or used to offer stronger models at the same price. Whether a closed API model is sparse is often undisclosed, so measured price and latency remain the practical signals.
Multilingual and multi-domain models
Problem: One model must handle many languages, code and specialist vocabulary without becoming too expensive to run. How it's applied: Expert capacity gives the model space to represent varied inputs while each token uses only a small slice. Outcome: Broad coverage without paying full-model compute for every token, though how experts actually divide the work varies by model and is not always intuitive. Teams relying on a model for a less common language or domain should still test it directly rather than assume an expert covers it well.
MoE vs. Dense Models
Neither architecture is strictly better; they make different trade-offs.
| Dimension | Dense model | Mixture of Experts |
|---|---|---|
| Compute per token | Full model activates every time | Only the selected experts activate |
| Parameter efficiency | 1 parameter set, always used | Many parameter sets, mostly idle per token |
| Memory / storage footprint | Matches active compute | Much larger than active compute (all experts must be loaded) |
| Training complexity | Standard backprop | Backprop plus routing + load-balancing losses |
| Inference latency at low batch size | Predictable | Can be less predictable — depends on routing and hardware layout |
| Best suited for | Smaller models, simpler serving infrastructure | Large-scale models where inference cost matters more than raw parameter count |
| Failure modes | Underfitting/overfitting, standard scaling limits | Routing collapse, expert underutilization, harder debugging |
The practical upshot: MoE wins when you're trying to maximize capability per unit of inference compute at large scale. Dense wins when simplicity, predictable latency, and straightforward deployment matter more than squeezing out extra capacity.
What This Means for Businesses and Builders
Most companies building on top of AI models never train an MoE model themselves — that's the labs' job — but the architecture still shapes decisions further down the stack.
Cost and latency planning. If you're comparing two models with similar benchmark scores, checking whether one is MoE and looking at its active-parameter count can tell you more about expected inference cost and latency than the total parameter count does. A "236B" MoE model that only activates 21B parameters per token may be cheaper and faster to serve than a 70B dense model, despite having a much bigger number in its name.
Hosting and infrastructure. For teams self-hosting open-weight models, MoE architectures have a specific catch: even though compute per token is low, you still need enough memory to hold every expert, because you don't know in advance which ones a given token will need. This means MoE models often demand more GPU memory or multi-GPU setups than their active-parameter count would suggest, even though the compute itself is lighter. Deployment planning has to account for both numbers, not just the one that sounds more favorable.
Fine-tuning behavior. Fine-tuning an MoE model can behave differently from fine-tuning a dense one, since gradients flow unevenly across experts depending on how the router assigns training examples. Teams doing domain adaptation on open MoE models sometimes find results less predictable than with dense models of similar size, and may need to pay closer attention to routing statistics during training.
Vendor evaluation. When evaluating API-based models where the architecture isn't disclosed, the practical signal that matters is still the same as ever: measured latency, cost per token, and quality on your actual workload. Architecture explains why a model might be cheap or fast, but it's not a substitute for benchmarking it directly for your use case — the kind of groundwork dedicated LLM integration work usually starts with.
Common Mixture of Experts Mistakes
Teams choosing, hosting or fine-tuning MoE models tend to trip over the same misunderstandings of what the two parameter counts mean.
Sizing hardware from active parameters
Active parameters describe compute, not memory. A team that provisions GPUs for a "21B active" model as if it were a 21B dense model will find it cannot even load the weights. Every expert must sit in memory, so hardware planning starts from total parameters, adjusted for quantisation.
Comparing models by total parameter count
The opposite error is treating a large total figure as a proxy for cost or quality. A "236B" MoE model may be cheaper to serve than a 70B dense model, and a bigger total does not guarantee better answers on your task. Compare active parameters for cost and benchmark quality directly.
Benchmarking latency at the wrong batch size
MoE routing and expert placement behave differently at low and high concurrency. Measuring latency with a single request, then deploying at high load, or the reverse, gives misleading numbers. Test at the batch sizes and traffic patterns you expect in production.
Fine-tuning as if it were a dense model
Fine-tuning data is routed unevenly, so some experts may barely change while others are overtrained, and routing itself can shift. Teams that skip monitoring per-expert utilisation sometimes get unpredictable results without knowing why. Track routing statistics during and after training, and compare results on held-out data against the base model.
Believing experts map neatly to topics
Marketing and explainers often describe "the coding expert" or "the maths expert". Analyses of trained models frequently find specialisation by syntax, position or token type instead. Building plans around assumed topic experts, such as pruning "unneeded" ones, can damage the model in unexpected ways. Only act on expert-level claims backed by analysis of that specific model.
Mixture of Experts Best Practices
Most teams will never train an MoE model, but many will choose, host or fine-tune one. The practices below keep the two parameter counts in their proper roles and replace architecture assumptions with measurements on your own workload. A short checklist for teams selecting between model options:
- Compare active-parameter counts, not just total parameter counts, when estimating inference cost.
- Check memory requirements separately from compute requirements if self-hosting — MoE models need more of the former relative to the latter.
- Benchmark latency at your actual expected batch size; MoE routing overhead can behave differently at low vs. high concurrency.
- If fine-tuning, monitor per-expert utilization rather than assuming training will behave like a dense model.
- Don't treat "MoE" as an automatic quality signal — it's a cost/capacity trade-off, not a guarantee of better outputs.
- Consider quantisation and expert offloading when memory is tight, and measure the speed and quality cost on your workload before committing.
- For multi-GPU hosting, use serving frameworks with explicit MoE support, and check how they place experts across devices, since communication overhead can erase the compute savings.
- Re-run your evaluation set whenever you switch between dense and MoE models of similar quality, because behaviour on edge cases can differ even when headline benchmarks match.
- Track cost per request and memory headroom in production, not just at launch, since traffic patterns and context lengths drift over time and change how sparse models perform.
- Document why a given model was chosen, including both parameter counts, the hardware it needs and the benchmark results on your tasks, so the decision can be revisited when new models appear.
- When an API provider does not disclose architecture, run a short load test at realistic concurrency and context lengths before committing, since price per token alone does not reveal how latency behaves under your traffic.
Real Limitations and Open Questions
MoE is a genuinely useful engineering trade-off, not a free lunch, and it comes with real costs that are easy to gloss over.
- Memory overhead. As noted above, you have to store and often load every expert even though only a few are used per token. On memory-constrained hardware, this can erase some of the compute savings the architecture is supposed to provide.
- Communication cost at scale. When experts are distributed across multiple devices (common for the largest models), routing tokens to the right expert means shuffling data across the network at every MoE layer. This inter-device communication can become a bottleneck that partially offsets the compute savings, especially at large batch sizes.
- Training instability. The load-balancing loss, the discrete routing decisions, and the interaction between the two make MoE models more finicky to train than dense ones. Getting routing to behave well — neither collapsing to a few experts nor scattering so evenly that specialization never emerges — is still more art than science.
- Interpretability. It's tempting to describe experts as specializing in topics ("the coding expert," "the math expert"), but empirical studies of trained models often find specialization along shallower or less intuitive lines. Treat any specific claim about what an individual expert "does" with skepticism unless it's backed by direct analysis of that model.
- Uneven quality per token. Because routing decisions are made independently per token, a model's effective capacity can vary slightly from token to token depending on which experts get selected — a subtlety dense models don't have, since every token always gets the full model.
None of this means MoE is a bad trade — the fact that it's now widespread among large model releases is evidence it works well enough in practice. But "sparse activation" is not synonymous with "solved problem," and teams building serious infrastructure around these models should expect some of the rough edges above.
There's also an open research question about how far the approach scales. Adding more experts increases total capacity, but each additional expert also adds memory overhead, communication cost, and another dimension for the router to get wrong. At some point the marginal benefit of another expert has to be weighed against the marginal cost of storing and coordinating it — and where that point sits isn't a fixed number, it depends on the hardware, the task mix, and the routing strategy in use. Labs experimenting with hundreds of small experts per layer are effectively testing where that ceiling is.
What to Watch Next
The MoE space is still evolving on a few fronts worth tracking:
- Finer-grained expert design. Some newer architectures split experts into smaller, more numerous units with more flexible routing (sometimes combined with a small number of "shared" experts every token always uses), aiming to get more benefit from specialization without as much routing instability.
- Better load-balancing techniques. Research continues into routing strategies that reduce the need for auxiliary losses that fight against the main training objective, since every extra loss term is a trade-off against pure task performance.
- Hardware co-design. As MoE becomes standard, expect more inference hardware and serving frameworks built specifically around sparse, conditional computation — reducing the memory and communication penalties that currently offset some of its compute savings.
- Hybrid approaches. Expect continued experimentation with hybrid architectures that mix dense and sparse layers, or that vary the number of active experts dynamically based on task difficulty, rather than treating "dense vs. MoE" as a binary choice.
Teams weighing architecture trade-offs for their own AI infrastructure, whether that means choosing between hosted models or planning a self-hosted deployment, can get hands-on help from Woyce Technologies.
FAQ
What does "mixture of experts" mean in AI?
It's a neural network architecture where, instead of one large feed-forward block processing every input, there are multiple smaller "expert" blocks and a routing network that sends each token to only a few of them. This lets the model have a large total capacity while keeping the compute cost per token low.
Is Mixture of Experts the same as an ensemble of models?
No. An ensemble runs multiple full models and combines their outputs, which increases compute cost. MoE runs one model but only activates a fraction of its parameters per token, which is designed specifically to keep compute cost down while increasing total capacity. The experts in an MoE layer are also trained jointly with a shared router inside a single network, whereas ensemble members are usually trained separately and only combined at the output.
Why do MoE models need more memory even though they're cheaper to run?
Because the router decides which experts to use on a token-by-token basis, the system generally needs every expert loaded and ready in memory, even though most sit idle for any given token. Compute is sparse, but storage isn't. That is why a model with a modest active-parameter count can still need several high-memory GPUs when self-hosted. Quantization and expert offloading can reduce the footprint, usually at some cost to speed or quality.
Does Mixture of Experts make models smarter, or just cheaper?
Both, in a sense. It doesn't make a fixed amount of compute produce better results by itself, but it lets you increase total model capacity (more experts, more parameters) without a proportional increase in the compute cost of using that capacity — which in practice has correlated with quality gains on many benchmarks.
How do I know if a model I'm using is a Mixture of Experts model?
Check the model card or technical report — MoE models are typically described with two parameter counts, like "total parameters" and "active parameters per token." If you only see one number, or if the documentation explicitly says "dense," it's using the standard architecture. Names such as "8x7B" are another hint. For closed API models the architecture is often undisclosed, in which case measured latency and price per token are the practical signals.
Are MoE models harder to fine-tune?
They can be. Training examples get routed unevenly across experts, which can produce less predictable fine-tuning behavior than with dense models. Some teams monitor per-expert utilization during fine-tuning to catch cases where certain experts are being over- or under-trained. Parameter-efficient methods such as LoRA are commonly applied to MoE models too, but test carefully on held-out data and watch for experts that stop being selected after training.
Will Mixture of Experts eventually replace dense models entirely?
Unlikely in an absolute sense — dense models remain simpler to train, deploy, and reason about, which still matters for smaller models or latency-sensitive applications with tight infrastructure constraints. MoE is best understood as the default choice once model scale reaches a point where inference cost becomes the binding constraint, not as a universal replacement.
Conclusion
Mixture of Experts solves a specific economic problem: model quality keeps improving with scale, but serving a dense model means paying for every parameter on every token. By replacing one large feed-forward block with many experts and a learned router that activates only a few per token, MoE lets labs grow total capacity while keeping per-token compute closer to that of a much smaller model.
The trade-off is not free. Every expert still has to sit in memory, distributed experts add communication overhead, routing needs load-balancing to avoid collapse, and training and fine-tuning are less predictable than with dense models. Claims that individual experts map neatly to topics deserve skepticism.
For teams building on these models, the practical rules are simple: estimate cost and latency from active parameters, size hardware from total parameters, benchmark at your real batch sizes, and judge any model on your own workload rather than its architecture label. If you are choosing between hosted and self-hosted models and want help sizing the decision, talk to our LLM integration team.