Mixture-of-Experts (MoE) models route each token to a few of many experts, yet what an individual expert is responsible for remains unsettled.
Existing analyses count how often an expert is selected on domain-labeled corpora, an instrument that:
- fixes the taxonomy in advance,
- stays coarse, and
- records selection rather than contribution.
We instead study specialization in feature space by training a sparse autoencoder (SAE) on the output of an MoE layer.
Since this output is a router-weighted sum of expert outputs and the SAE encoder is affine, the pre-activation of every latent decomposes exactly over the experts.
Aggregated over a corpus, this gives each expert a dominant set of latents whose activation it principally drives.
实验结果
On layers 4, 30 and 46 of Qwen3-30B-A3B, contribution is far more concentrated than gating, most latents have only one or two dominators, and the overlap between dominant sets concentrates on a few expert pairs.
- Java and front-end web code
- geometry versus arithmetic and polynomials
while the coarse domains themselves take very unequal numbers of experts.
资源
Our SAEs and code are available at https://github.com/Bytedance-Seed/SAE-Interprets-MoE.