Hybrid models combine softmax attention with linear attention, which compresses history into fixed-size recurrent states to balance model quality and inference efficiency.
However, linear attention has not pushed this compression to its limit: we find that the
- queries
- keys
- values
Yet existing serving systems keep these redundant states dense, reading and rewriting them in full at every decoding step, so their memory traffic grows with the batch size and dominates the decoding cost.
Thus SinkState stores each recurrent state in an adaptive low-rank form throughout inference with
- buffered updates
- a local SVD restricted to the top subspace
SinkState further consolidates the remaining components asynchronously on the CPU.
On four hybrid models with
- Mamba-2
- Gated DeltaNet (GDN) layers
- 4B
- 120B parameters
- 2.6x (1.9x on average)
- 1.58x (1.25x on average)
- Mamba-SSM
- FLA