Readout scaling changes what a sparse autoencoder (SAE) learns, but the benefit of removing it depends on the data.
We use readout-canonicalized decision-space (RC-DS) coordinates to remove singular-value gains while preserving retained decision information.
Experiment Details
At a fixed readout condition number, changing the generating distribution reverses which training coordinates recover known factors best.
In our controlled coordinate-matching study, recovery is highest when training restores the space in which the factors were generated.
When factors are generated before readout scaling, RC-DS increases mean sparse-factor recovery from 0.506 to 0.913 at condition number 100, compared with centered-logit training under the same budget.
An exact wrapper reproduces RC-DS training to numerical precision, isolating training geometry rather than additional information.
Neural Policies
In neural policies, RC-DS improves mean logit reconstruction under the tested budgets.
On three fixed language-model checkpoints, RC-DS has higher mean gain-free reconstruction in all 18 tested task-sparsity settings, while centered-logit SAEs reconstruct output logits better in 17.
Small-label-head Controls
Small-label-head controls show that longer training reduces the reconstruction gap from 0.0198 to 0.0016.
Conclusion
Together, these results show how readout scaling, the data distribution, and training shape sparse learning.
RC-DS provides an information-preserving method for studying and controlling the readout's contribution.
Separate label-detection and steering tests distinguish these reconstruction effects from task-specific feature usefulness.