动机与观察
Activation sparsity can reduce FFN weight traffic during single-user LLM decoding, but active channels must be identified early enough to avoid fetching unused weights.
In pretrained SwiGLU models, training-free routing with the native gate performs substantially worse than routing with the up activation at high sparsity.
We show that this ordering reverses after sparsity-aware training.
SGA-Sparse 方法
Based on this insight, we introduce SGA-Sparse, which converts pretrained SwiGLU LLMs into fixed-budget activation-sparse models by training the existing gate projection directly as a top-k router.
SGA-Sparse updates all model parameters end-to-end while preserving the original SwiGLU activation and introducing no auxiliary router parameters.
训练结果
After approximately 30B training tokens, LLaMA models from 1B to 8B achieve uniform per-token 75–85% FFN channel sparsity while remaining within 0.2–2.7 pp of their original dense checkpoints in average zero-shot accuracy.
量化与 QAT
Because post-training quantization degrades sparsity-trained checkpoints, we further study two complementary quantization-aware training (QAT) recipes: continued-QAT largely recovers performance at 4-bit weight quantization, while joint-QAT performs better for the more aggressive sub-4-bit target.
与稠密低比特的比较
To compare activation sparsity with dense low-bit quantization, we translate the sparsity-induced reduction in weight traffic into an equivalent bit-width.
Under this unified weight-traffic metric, our 2.5- and 1.5-bit-equivalent models establish a new accuracy–traffic Pareto frontier: the 2.5-bit-equivalent models match dense 3-bit QAT accuracy, while the 1.5-bit-equivalent models approach dense 2-bit models and outperform dense ternary models by 1.3 pp on average.
结论
These results establish native-gate training as a practical approach for combining high FFN sparsity with low-bit weights.