背景与研究动机
In Mixture-of-Experts (MoE) models, the nature of expert specialization remains controversial. Reports of domain-associated expert preferences contrast with analyses attributing routing primarily to token identity or syntactic patterns.
To clarify these differing accounts, we look beyond expert-level routing to investigate the neuron-level organization of domain-specialized computation using Routing-Aware Magnitude Profiling (RAMP), a measure developed for this analysis.
主要发现：领域特化与训练演化
Using mixed-domain calibration data, we identify domain-specialized neuron sets with 58–74% overlap across multi-source samples of the same domain, compared with 15–24% across domains.
To understand how this organization evolves during training, we trace neuron importance throughout pre-training and post-training across three fully open MoE families. Neuron importance continues to reorganize between pre-training checkpoints despite comparatively stable expert-level importance rankings, while post-training largely preserves neuron rankings.
方法与结果：RAMP-based Pruning (RAMPr)
Building on these neuron-level perspectives, we further introduce RAMP-based Pruning (RAMPr) to select neurons across experts for balanced multi-domain capability preservation.
- At 75% neuron retention, RAMPr preserves average 91–98% of original full performance across six models.
- At 50% retention, our pruning method can reduce resident GPU memory by 36–43% and improves batched prefill throughput by 17–39% effectively.
可用资源
Our code is available at https://github.com/ydyjya/Neuron-Level-Specialization-in-MoE