Agent Memory Below the Prompt: Persistent Q4 KV Cache for Multi-Agent LLM Inference on Edge Devices

Researchers developed a memory management system for multi-agent AI systems on edge devices that reduces memory requirements by 4x through 4-bit quantization and eliminates redundant computation by persisting KV caches to disk. The solution reduces time-to-first-token by up to 136x while maintaining minimal impact on model quality across three major language model architectures.

• →Edge devices can only fit 3 AI agents simultaneously due to memory constraints, requiring constant cache eviction and reload.
• →The new system uses 4-bit quantized KV cache persistence to disk, reducing memory requirements by 4x compared to FP16.
• →Time-to-first-token improved by 3-136x across Gemma, DeepSeek, and Llama models at various context lengths.
• →Quality impact is minimal with perplexity changes ranging from -0.7% to +3.0% across tested architectures.
• →The solution enables efficient multi-agent workflows on resource-constrained edge devices without redundant computation.