SIFT: Selective-Index For Fast Compute of RAG Prefill by Exploiting Attention Invariance

Researchers introduce SIFT, a novel optimization technique for Retrieval-Augmented Generation (RAG) systems that exploits attention patterns to accelerate LLM prefill computation. By storing only compact bit vectors of high-attention locations rather than full KV tensors, SIFT achieves 1.71x faster time-to-first-token while reducing storage by up to 24,000x and maintaining accuracy within 1% of standard methods.

SIFT addresses a critical bottleneck in RAG systems where injecting relevant documents into LLM prompts significantly increases latency during prefill—the initial phase before generating the first output token. Current approaches either recompute all tokens inefficiently or precompute key-value tensors that require expensive disk transfers, both degrading performance on modern GPUs. The research identifies two pivotal attention invariance properties: local-attention remains consistent within documents regardless of context, and high-attention keys attract cross-attention from subsequent documents. This enables SIFT to predict attention patterns without full recomputation.

The practical implications are substantial. By storing only selective attention index locations as bit vectors instead of complete KV tensors, SIFT dramatically reduces memory footprint and eliminates disk I/O bottlenecks. The 1.71x speedup in time-to-first-token directly improves user experience in production RAG systems, particularly for document-heavy applications in enterprise search, customer support, and knowledge retrieval. The minimal accuracy degradation (within 1%) makes this a viable production solution.

This optimization becomes increasingly valuable as LLMs scale and RAG adoption grows across industries. The work represents incremental but meaningful progress in LLM inference efficiency—a critical competitive factor as organizations deploy these systems at scale. The approach is generalizable across different model architectures, suggesting broader applicability. However, the real-world impact depends on integration into existing inference frameworks and widespread adoption by practitioners.

• →SIFT exploits attention invariance properties to reduce RAG prefill latency by 1.71x while maintaining accuracy within 1% of baseline
• →Storage overhead drops from full KV tensors to compact bit vectors, reducing requirements by up to 24,000x and eliminating costly disk transfers
• →Local-attention invariance and cross-attention consistency enable fine-grained prediction of high-attention locations without full recomputation
• →The method stores no actual KV data, addressing the primary performance limitation of existing KV-caching approaches on modern GPUs
• →Production deployment potential is high given the combination of significant speedup, minimal accuracy loss, and architectural generalizability