Since GPT, most Transformers have repeated the same attention mechanism at every layer. Yet this design is largely a convention rather than a tested conclusion.
When multiple sequence mixers are combined in one stack, improvements may arise from mechanism choice, placement, or both, making causal attribution difficult.
We introduce Aether-7B-5Attn
We introduce Aether-7B-5Attn, a 6.59B-parameter mixture-of-experts model ($\approx$2.98B active) whose 49 layers contain seven sequence-mixing mechanisms arranged as a $7\times7$ Latin square.
Because each mechanism appears exactly once in every row and column, the design guarantees balanced exposure across depth while eliminating placement confounds.
To evaluate this principle, we build a parameter-matched proxy with four mechanisms arranged as a $4\times4$ Latin square over sixteen layers, matched to 700.9M parameters and trained with eight seeds per arm.
The results reveal a clear dissociation.
Rearranging a distributed heterogeneous stack into a balanced periodic cycle changes validation loss by only 0.16%, indicating that exact placement has little effect.
In contrast, clustering the same mechanisms into contiguous depth bands incurs a 0.59% penalty, while replacing the heterogeneous stack with a homogeneous one incurs a 1.68% penalty.
These results indicate that performance depends primarily on heterogeneous composition distributed across depth rather than on any particular permutation.
We confirm this finding at 2.16$\times$ larger scale (1.514B parameters), where the homogeneous-stack penalty increases to 2.63% and removing the SSM-family mechanism produces a 3.20% degradation.
We further report per-mechanism cost profiles, English and Korean evaluations, and a causal-safety audit of all 49 layers.
We release model weights, training recipes, training code, logs, and architecture source code.