Mixture of ExpertsMoE
A model architecture that activates only a fraction of its parameters per request.
Mixture of Experts (MoE) is an architecture where the model contains many independent sub-networks ("experts") but only routes each input token through a small subset of them — typically 2–8 out of dozens. A learned router decides which experts to activate.
Why it matters: MoE allows massive total parameter counts (= more knowledge) while keeping inference compute manageable. Mistral's Mixtral 8×7B activates 12B parameters per forward pass despite having 47B total. Grok v9 Medium has 1.5T total but far fewer active parameters.
Tradeoffs:
- Pro: More total knowledge per FLOP spent during inference
- Pro: Can specialise different experts for different domains
- Con: Requires more memory to hold all expert weights (even those not active)
- Con: Load balancing across experts adds engineering complexity
Models using MoE: Mixtral 8x7B/8x22B, Grok 1/2/3, Gemini 1.5 (claimed), GPT-4 (rumoured), Qwen3-235B.
In plain terms
A hospital where a triage nurse (router) decides which specialist to send each patient to. The hospital employs 20 specialists but only 2–3 see any given patient.