Doubling the active experts on a frozen 35B checkpoint, then healing the resulting mixture with a residual-delta expert fine-tune. Same weights, wider router.
The only structural edit is the router's top-k. Everything downstream reacts to it.
The router computes a softmax over all 256 experts in fp32, keeps the top-k, and
renormalizes the selected weights unconditionally — renormalization is baked into the
architecture, so going from k=8 to k=32 silently redistributes the entire gate mass across
four times as many experts. Because the added ranks 9–32 carry 54.0% of that
renormalized mass, they are load-bearing, not tail noise. Experts are stored as packed 3D tensors
(gate_up_proj [256, 1024, 2048], down_proj [256, 2048, 512] per layer),
so widening k adds compute but zero new parameters.
Measured risk — naive k=32 without retraining. Flipping top-k alone degrades knowledge benchmarks: MMLU 0.843 → 0.807 and GSM8K 0.893 → 0.865 (n=600). The 24 added experts per token were never trained to collaborate inside the mix — an uncalibrated mixture. That is exactly what the tuning recipe below repairs.
Roughly 2× active compute per token, at zero weight cost.
Widening the router turns latent capacity that already lives in the checkpoint into active compute: ≈3B → ≈6.6B active parameters per token with no new weights to store or serve. The working hypothesis is a complexity gradient — complex, multi-step work (agentic terminal use, hard coding) should benefit most from the extra mixture capacity, while simple factual recall benefits least. So the payoff and the risk sit on different axes: reasoning throughput goes up, and knowledge is the axis we must actively protect.
Coding (post-train)
0.902
HumanEval vs stock-k8 0.866 · n=164, trend
Knowledge (to heal)
−2.0 pt
MMLU vs stock · statistical tie, p=0.14
Active compute
≈2×
≈6.6B active/token · +0 weights
Gains grow with task complexity; the knowledge gap is currently a statistical tie rather than a clean win, so MMLU is the axis being actively healed rather than a settled result.
Profile the wider router, then teach only the experts it leans on to collaborate.
Run the target corpus through the model at k=32 and count token-level routing frequency for every expert.
Keep experts by cumulative routing frequency up to top-p 0.2 — the experts that actually do the work.
Train only delta tensors added to the selected experts' FFN slices. The router and every other weight stay frozen. Deltas are toggleable — patched on or off at load time.
Mixed corpus, 428M tokens, contamination-gated to zero residual hits against every eval benchmark. A small general + knowledge replay slice is deliberate forgetting insurance.
8×GPU DDP at sequence length 7168, with fused linear cross-entropy (no logits materialization) and Adafactor.
The routing distribution is the coordinate system the deltas are trained in.
The deltas are learned relative to a specific routing distribution. Freeze it and every delta keeps meaning the same thing at inference; move it and the ground shifts underneath them.
Letting the router move under SFT/RL is the classic path to routing collapse, where a few experts capture all traffic. A frozen router removes that failure mode entirely.
Because nothing but the deltas changed, turning the delta off returns the exact stock model. The base checkpoint is never at risk.
From supervised mixture-healing toward Terminal-Bench.
Done
Residual-delta ESFT on the 428M mixed corpus.
Next
Retrain on self-generated, verified solutions.
Then
Rule-based patch-similarity reward.
Target
The north-star agentic benchmark.