QCFuse: Query-Aware Cache Fusion via Compressed View for Efficient RAG Serving

QCFuse introduces a compressed-view query-aware selector for retrieval-augmented generation (RAG) systems that accelerates LLM serving by intelligently reusing cached key-value computations. The technique achieves 1.7x speedup over full prefill and 1.5x over existing baselines while maintaining full-prefill quality, addressing a critical bottleneck in RAG deployment.

QCFuse addresses a fundamental efficiency problem in RAG systems: the prefill stage, where retrieved context is processed alongside user queries, represents a dominant serving cost. Traditional cache fusion approaches force developers to choose between two suboptimal strategies—fast but quality-compromising query-agnostic selectors, or thorough but pipeline-blocking full-view selectors that require inspecting all layers before recomputation begins.

The innovation lies in QCFuse's compressed-view approach, which uses chunk-anchor query probing and critical-layer profiling to identify which cached tokens can be reused and which require recomputation. This design eliminates the need for complete layer visibility, allowing the system to operate efficiently within the layer-wise cache-fusion pipeline without sacrificing answer quality.

For the AI infrastructure ecosystem, this work matters significantly. RAG has become essential for production LLM applications seeking better factual grounding and reduced hallucination. However, serving costs directly impact deployment feasibility and economic viability. Achieving 1.7x prefill speedups while maintaining quality translates to meaningful cost reductions for enterprises running inference at scale, potentially enabling wider RAG adoption and reducing operational expenditure for AI service providers.

The implementation in SGLang and evaluation across four open-weight models and six datasets provides practical validation. As RAG systems proliferate in production environments, optimizations that preserve quality while reducing latency and compute requirements will become increasingly valuable. The focus on efficient serving aligns with broader industry trends toward cost-effective inference and makes QCFuse relevant for practitioners deploying LLMs in resource-constrained or cost-sensitive scenarios.

• →QCFuse achieves 1.7x prefill speedup over full prefill and 1.5x over ProphetKV baseline while maintaining full-quality answers
• →Compressed-view query-aware selection eliminates the efficiency-quality tradeoff that hampers existing RAG cache fusion approaches
• →Critical-layer profiling enables selective token recomputation without requiring all-layer inspection, preventing pipeline stalls
• →The technique is validated across four open-weight LLMs on six datasets, demonstrating broad applicability
• →Significant cost reduction potential for enterprises running inference-heavy RAG systems at production scale