Five Architectures, One Shared Space Cross-Model Representational Alignment Without Weight Merging

Combining language models is usually done by averaging weights, which works only when the models share a common starting point.

We ask whether five frozen, independently trained models with different internal designs (four transformers from four organizations and one state-space model) can be related through their activations alone, with no weight merging.

研究问题与初步发现

Using permutation tests, an independent replication probe set, depth sweeps, pooling ablations and confound checks, we find statistically robust correspondence between how the five models organize the same text. It is strongest at each model's final layer and weaker for the state-space model, whose weakest pairing is still 35-43 times chance.

构建共享空间与实验结果

We use this to build a shared space with Generalized Procrustes Analysis, in which no model is a fixed reference.

At the final layer, mean held-out retrieval accuracy over the twenty ordered model pairs is:

• 0.91 to 0.93 on two probe sets (chance 0.03)
• beating pairwise linear maps (0.82 to 0.85)

However, the space is worse than those maps in the middle layers.

适用性与局限性

It transfers to ordinary English of unfamiliar grammatical form and fails on mathematical and logical notation.

路由测试与负面结果

We then test whether the space supports a router that picks among the five models on gradeable tasks.

Across four task types we find no case where routing beats always using the strongest model, and we report this as a negative result.

A correctness signal that looked strong turned out to be category membership in disguise.

All experiments ran on one laptop without a GPU.