Indexer-based sparse attention reduces the cost of core attention by passing only a fixed, small number of important tokens to it.
However, the indexer must still score the entire KV cache at every decoding step.
This scoring overhead becomes a major bottleneck as the context length grows.
We propose SPIN (Shadow Predictive Indexer) to reduce this indexer overhead.
SPIN uses lightweight, history-based prediction to identify important KV blocks, avoiding the need to score the full KV cache at every decoding step.
SPIN treats KV blocks and speculative decoding as first-class design and implementation considerations.
Performance Evaluation
Across extensive evaluations on long-context and agentic benchmarks, SPIN achieves 30-40% sparsity while preserving task quality.
In end-to-end vLLM serving, SPIN improves output throughput by up to 14.9% and reduces median inter-token latency by up to 13.2%.