End-to-end speech large language models (LLMs) often answer the same question less accurately from speech than from text, even when the spoken content matches the text prompt.
Existing analyses study this modality gap through the geometry of representations and measure how closely speech representations align with text.
Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way.
This work connects representation geometry with answer formation and asks whether geometric alignment implies functional equivalence.
Experiment Design
We compare cross-layer Centered Kernel Alignment (CKA) with layer-wise answer margins for four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH.
Qwen2.5-Omni and MiniCPM-o
In Qwen2.5-Omni and MiniCPM-o, speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer alignment does not imply functional equivalence.
Qwen2-Audio
Qwen2-Audio instead shows weak alignment, which indicates a different failure mode.
Attention Intervention
An attention intervention at fixed input and token count identifies a causal effect of late attention allocation on answer-code separation, a quantity distinct from answer correctness.
Speech Signal Compression
Compressing the speech signal or its token sequence repairs a subset of the errors that occur only with speech.