RelayCaching: Accelerating LLM Collaboration via Decoding KV Cache Reuse

Researchers introduce RelayCaching, a training-free method that accelerates multi-agent LLM systems by reusing KV cache data from previous agents to eliminate redundant computation. The technique achieves over 80% cache reuse and reduces time-to-first-token by up to 4.7x while maintaining accuracy across mathematical reasoning, knowledge tasks, and code generation.

• →RelayCaching enables direct reuse of KV caches from previous agents in multi-agent LLM systems without requiring additional training.
• →The method achieves over 80% KV cache reuse rates by selectively recomputing only sparse, localized deviations in specific layers and token positions.
• →Performance improvements include up to 4.7x reduction in time-to-first-token compared to standard pipelines with negligible accuracy loss.
• →The approach addresses a critical bottleneck in collaborative AI systems where redundant prefill computation significantly increases memory usage.
• →Experiments demonstrate effectiveness across diverse tasks including mathematical reasoning, general knowledge queries, and code generation.