Sparse autoencoders (SAEs) are commonly evaluated by variance explained and ablation damage. We show how massive activations undermine both evaluations, and propose a correction.
Massive activations cause up to 99.9% of activation variance to lie along the first principal component (PC1).
- Because the PC1 component has no unique sparse decomposition, each SAE run rebuilds it from a different set of features.
- These features dominate importance rankings: an average of 47 of the 50 most important features are unmatched in a separate run, compared with about 27 of a random 50.
- A dictionary can also report 99.96% of variance explained while leaving 33% of the variance outside PC1 unexplained.
We bypass PC1 by training the SAE on activations with PC1 removed, then adding PC1 to the reconstruction.
In affected models, the top-50 feature sharing rate rises from 4-39.5% to 73-99.5%. At the same TopK sparsity, the resulting SAE degrades the model up to 44% less and reduces content reconstruction error by up to 33.9%.