ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision.
Technical Mechanism
It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states.
Core Features
- As a highly plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation.
- It also supports high-dimensional position embeddings via high-dimensional convolutions.
Experimental Results and Availability
In a reproduction, ConvPE outperforms five 1D baselines and eight 2D baselines.
The code is publicly available at:
https://github.com/wzhagi/convpe