EXHIBIT 04 / SPARSE EXPERTS
How can a 671B-parameter model avoid using every parameter for each token?
In a sparse mixture-of-experts layer, a router scores experts for each token and selects a small subset. The unselected expert parameters still belong to the model; they simply do not participate in that token’s expert computation.
THE SHORT ANSWERScore. Select. Compute. Combine.
6–10 MIN · STEP-BY-STEP · NO ACCOUNTA token takes a route through the expert layer.
- 01 / ROUTERScore the experts
A learned router produces a score for each routing expert for the current token.
- 02 / TOP-KSelect a few
Only the highest-scored routing experts are chosen, alongside any experts defined as shared.
- 03 / EXPERTSRun the selected blocks
Selected experts transform the token representation. Unselected routing experts do not run for that token.
- 04 / COMBINEMerge their outputs
The selected outputs are weighted and combined before the model continues.
The DeepSeek-V3 abstract reports 671B total parameters and 37B activated per token for the whole model. The control below models one equal-sized expert layer, so its numbers are not a reconstruction of that 671B/37B account.
One token can choose a narrow path—and still create a crowd.
The expert identities and scores are a teaching trace. The load experiment below is a separate real program with fixed random seeds.
TOKEN
A representation arrives
The router receives the current token representation at an expert layer.
The route is chosen per token, not once per topic.
Keyboard: focus this record and use ← or →. Nothing advances automatically.
Change the route. Keep the two parameter accounts separate.
The browser uses the same exact-integer kernel as the Chinese exhibit. Every expert is assigned an illustrative 2B parameters; one shared expert is always selected.
- Selected share of this layer’s expert parameters
- 3.5%
- Routing experts not selected this time
- 248
- Illustrative size of each expert
- 2B parameters
Each token selects 8 routing experts plus 1 shared expert, or 3.5% of this illustrative layer’s expert parameters.
Both bars are parameter counts for one illustrative expert layer. They exclude attention and other non-expert layers, and they do not measure memory, operations, communication, or latency.
This control counts parameters only. Real memory, operations, communication, latency, model quality, and the right value of top-k depend on implementation and training; none is measured here.
A real router simulation measures imbalance, not model quality.
Average share was 62.5 across 256 experts in one fixed 2,000-token toy run.
Share of 16,000 selections taken by the busiest 64 experts; uniform share would be 25.0%.
Busiest expert in the fixed run; across eight seeds the range was 85–106.
A deterministic toy router
The Python script routes 2,000 generated token vectors across 256 experts with top-k = 8, with and without a small balancing intervention.
Selection counts
It records busiest and least-used experts, top-quartile share, and ranges across eight fixed seeds.
Every MoE behaves this way
Eight toy seeds do not establish how all trained MoE models route, how experts specialise, or how fast a deployment will be.
RECORDED RUN / EXECUTED
Inspect the result behind the interpretation.
Without intervention, the busiest expert received 257 selections versus an average share of 62.5. With the script’s minimal balancing rule, the busiest expert received 91.
Tracked content revision: 2e1ffedbde69022b2152f5bae897d724099cdf82892d5c6af6d6b636c3184cf4
Show exact lines from the original stdout
The program’s original labels are in Simplified Chinese; the lines and numbers below are copied without translation.
最忙的专家被选中 257 次 平均份额 62.5 次（4.1 倍）
最忙的专家被选中 257 次 平均份额 62.5 次（4.1 倍）
最忙的 64 个（25%）占了 49.3% 均匀分配时应为 25.0%
最忙的专家被选中 91 次 平均份额 62.5 次（1.5 倍）PRIMARY & PRACTICE SOURCES