SpotAttention: Plug-In Block-Sparse Routing for Pretrained Long-Context Transformers

SpotAttention is a lightweight machine learning technique that reduces computational costs for large language models processing long text sequences. By learning to identify only the most relevant tokens to attend to, it achieves 3.9x faster decoding speeds while maintaining accuracy at context lengths eight times longer than training, addressing a critical efficiency bottleneck in modern LLMs.

SpotAttention tackles one of the most pressing technical challenges in deploying large language models: the quadratic computational cost of attention mechanisms over long sequences. As LLMs increasingly handle contexts of 100K+ tokens, the prefill and decode stages become prohibitively expensive. This work introduces an elegant solution by training a lightweight selector module through knowledge distillation that learns to mimic the attention patterns of frozen pretrained models, avoiding the overhead of selecting which tokens to attend to.

The technique emerges within a broader trend of efficiency optimization in LLMs. Previous approaches like FlashAttention improved memory access patterns, while others explored sparse attention designs. SpotAttention distinguishes itself by remaining agnostic to the underlying model architecture—it works as a plug-in on top of existing transformers without retraining them. The dual top-p rule dynamically adjusts per-layer token selection based on calibrated confidence scores, enabling graceful adaptation across different computational budgets.

For practitioners and organizations, the results are significant: achieving 3.9x speedups over FlashAttention at 128K context lengths dramatically reduces inference costs and latency. The quantization findings—reducing selector cache size by 3.5x using INT4/FP4 formats—indicate the technique remains efficient even with aggressive compression. This matters for edge deployment, mobile applications, and large-scale serving infrastructure where inference costs directly impact profitability.

Key watching points include whether this approach generalizes across diverse model families beyond Qwen, whether the performance gaps narrow at even longer contexts, and whether similar distillation-based selectors become standard practice in production deployments.

• →SpotAttention reduces decode latency by 3.9x compared to FlashAttention while maintaining accuracy at 128K token contexts
• →The technique works as a plug-in module on frozen pretrained models, requiring no full retraining
• →Selector K-cache quantization to INT4/FP4 shrinks memory footprint by 3.5x without accuracy loss
• →Performance scales to context lengths eight times longer than training data without degradation
• →Distillation-based token selection offers a practical alternative to fixed sparse attention patterns