Cross-Layer Transcoders (CLTs) decompose the activations of large language models into human-interpretable features and trace how these features interact across layers to form circuits, making them a key tool for mechanistic interpretability. However, training CLTs effectively and efficiently is hard. Beyond the usual trade-off between sparsity and reconstruction quality, a CLT has to model all layers at once, which adds a further challenge.
现有挑战
- The common JumpReLU recipe controls sparsity only indirectly through a penalty.
- The direct extension of TopK selects a fixed number of latents in every layer.
- Yet reconstruction difficulty differs widely across layers, and intermediate layers are much harder than others, so giving every layer the same budget is suboptimal.
核心方案：Cross-Layer TopK
We propose Cross-Layer TopK, which sets a single L0 budget for the whole model and dynamically allocates it across layers during training.
实验结果与贡献
- In a comparison with JumpReLU and other TopK variants on Qwen3-8B-Base, this gives lower FVU while using 20% fewer active latents than JumpReLU.
- The better reconstruction carries over to circuit tracing, yielding more complete circuits with higher replacement and completeness scores.
- The fixed per-token budget of Cross-Layer TopK also makes it easy to pair with sparse decoding, which cuts both computation and communication and gives a 2.9x speedup per training step.
- We further show that Cross-Layer TopK follows scaling laws in both latent size and sparsity.
We have open-sourced the weights of our CLT at https://huggingface.co/ByteDance-Seed/Cross-Layer-TopK-CLT.