首页 > AI前沿 > SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

SinkState: On-the-Fly SVD of Recurrent States for Efficient Hybrid LLM Serving

arXiv机器学习 2026-11-10 23:57 20 阅读 查看原文

Hybrid models combine softmax attention with linear attention, which compresses history into fixed-size recurrent states to balance model quality and inference efficiency.

However, linear attention has not pushed this compression to its limit: we find that the

  • queries
  • keys
  • values
of linear layers concentrate in a stable, low-dimensional subspace, a phenomenon we call the linear attention sink, so each state is effectively low-rank, with a rank that varies across states.

Yet existing serving systems keep these redundant states dense, reading and rewriting them in full at every decoding step, so their memory traffic grows with the batch size and dominates the decoding cost.

Thus SinkState stores each recurrent state in an adaptive low-rank form throughout inference with

  • buffered updates
  • a local SVD restricted to the top subspace
.

SinkState further consolidates the remaining components asynchronously on the CPU.

On four hybrid models with

  • Mamba-2
  • Gated DeltaNet (GDN) layers
, from
  • 4B
  • 120B parameters
, SinkState reduces the peak memory of recurrent states by up to
  • 2.6x (1.9x on average)
and speeds up decoding by up to
  • 1.58x (1.25x on average)
over
  • Mamba-SSM
  • FLA
, while maintaining accuracy.