You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Researchers propose Cross-Layer Sparse Attention (CLSA), a novel architecture that optimizes long-context LLM inference by sharing both key-value caches and routing indices across decoder layers. The method achieves up to 7.6x decoding speedup and 17.1x throughput improvement at 128K context while maintaining accuracy, addressing the efficiency-quality tradeoff that has constrained existing sparse attention approaches.

The computational bottleneck of long-context language model inference represents a critical challenge for production LLM deployment, particularly as reasoning-heavy applications demand longer sequences and intermediate chain-of-thought generation. CLSA tackles this by building on KV-sharing architectures like YOCO and introducing a key innovation: computing token-level top-k selection once and reusing that routing decision across all decoder layers. This shared-indexing approach elegantly solves a fundamental inefficiency in token-sparse methods, where the cost of recalculating top-k selections for each layer negates the performance gains from selective attention.

The technical contribution addresses a real architectural constraint in modern transformers. Previous sparse attention methods fell into two camps: structured block-sparse approaches offered speed but sacrificed quality, while token-sparse methods preserved accuracy but remained computationally expensive due to repeated routing calculations. CLSA's amortization of routing overhead bridges this gap, maintaining fine-grained selectivity while achieving meaningful wall-clock improvements.

For practitioners deploying large models on resource-constrained infrastructure, these efficiency gains directly translate to reduced latency and lower computational costs during inference. The 17.1x throughput improvement at 128K context length is particularly significant for applications requiring sustained long-context reasoning, where inference efficiency becomes the primary cost driver. This work demonstrates that architectural innovations in attention mechanisms continue to yield substantial practical benefits beyond incremental improvements. The ability to handle longer contexts efficiently without quality degradation supports broader adoption of reasoning-capable models in production environments.

• →CLSA shares routing indices across decoder layers, eliminating redundant top-k computations while preserving token-level selectivity.
• →The architecture achieves 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context length.
• →This approach simultaneously improves pre-filling, KV-cache storage, and long-context decoding efficiency.
• →The method maintains accuracy comparable to dense attention while significantly reducing computational costs.
• →CLSA provides a more complete solution for long-context LLMs by jointly advancing model quality and inference efficiency.