首页 > AI前沿 > High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference

High-dimensional layer-state trajectories reveal the internal dynamics of transformer inference

arXiv机器学习 2026-07-15 08:02 5 阅读 查看原文

Mechanistic studies of large language models localize function in parameters, attention heads or neurons, but those parts do not specify how one inference unfolds.

We treat the ordered layer-state trajectory as the empirical process and test whether layer order defines a measurable computation.

Representative Checkpoints

Across representative Qwen, Llama and Gemma checkpoints, realized order produced paths 1.60--2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles.

Lexical Task

The order effect transferred to a lexical task, but was stronger in three random initializations of the same Qwen architecture, identifying an architecture-compatible scaffold rather than a training-specific law.

Hidden States and Coordinates

Direct hidden states provided stronger mechanism coordinates than vocabulary readbacks, and a task-defined coordinate made candidate competition computable.

Transition Actuators

After additive interventions failed, local transition actuators selectively controlled internal trajectories across checkpoints.

Confidence Gate

A confidence gate then drove 17 of 87 held-out Qwen closure trajectories across the full-vocabulary top-1 candidate boundary while leaving all 203 non-closure top-1 outputs unchanged.

Safety-Matched Action Reassignment

Safety-matched action reassignment performed similarly, locating the gain in selective gating and the bounded actuator family rather than fine-grained dose assignment.

Layer Order

Layer order exposes an observable, computable and behaviourally consequential process of transformer inference without establishing a complete theory of LLM dynamics or control of answer correctness.