If multilingual language model representations contain separable language-associated structure, then shifting between languages may require only simple activation-space interventions.
To test this hypothesis, we study multilingual in-context learning (ICL) as a controlled setting: few-shot demonstrations are provided in English while the test query is in a target language, creating a clear cross-lingual transfer challenge.
Proposed Method
We propose language vectors, which are computed as mean activation differences between parallel source and target language examples at a particular layer and applied as inference-time offsets to the model's hidden states.
Experiments
Across three tasks spanning 19 languages and three models, our method improves over English few-shot prompting without any parameter updates.
Interpretability
Beyond performance, the vectors encode interpretable linguistic structure: closely related languages form tight clusters, and vectors transfer effectively across tasks and languages, with closely related languages' vectors recovering more of the gain than distant ones.
Findings
Together, these findings suggest that language occupies structured directions in multilingual activation space, and that inference-time interventions on English demonstrations can narrow the cross-lingual ICL gap.