From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG

Researchers introduce EPIC, a novel approach to on-device Retrieval-Augmented Generation (RAG) that prioritizes user preferences as compact personal context while operating under strict memory constraints. The method achieves dramatic efficiency gains—reducing memory usage by 2,404x and latency by 32x—while improving preference-following accuracy by 18.79 percentage points across multiple benchmarks.

EPIC addresses a critical challenge in deploying personal AI agents on consumer devices: balancing privacy, responsiveness, and memory efficiency without sacrificing contextual awareness. Traditional RAG systems store extensive raw data and retrieve based on similarity metrics, which proves inefficient for personal devices with limited storage and the need to maintain user preference alignment. The innovation centers on treating user preferences as a stable, compressed representation of personal context rather than exhaustively indexing raw information.

The broader trend reflects the industry shift toward edge AI and on-device processing as privacy concerns intensify and regulatory frameworks tighten around data handling. Users increasingly demand AI assistants that understand their preferences without constant cloud synchronization. EPIC's approach—selective retention of preference-relevant information and alignment of retrieval mechanisms—represents a practical solution to this architectural challenge that scales across different device types.

For developers and device manufacturers, this work reduces deployment friction significantly. Maintaining sub-1MB memory footprints while supporting natural language queries opens opportunities for AI integration across resource-constrained devices. The streaming update capability handles preference drift, addressing a real-world requirement where user preferences evolve over time. This capability matters for practical deployment in applications like recommendation systems, personalized assistants, and adaptive interfaces.

The 5-29ms per-query latency across platforms suggests viable real-time responsiveness. Future developments will likely explore how preference-aligned indexing scales with multi-modal data and whether this approach generalizes beyond the tested benchmarks to less-structured personal contexts.

• →EPIC reduces on-device RAG memory footprint by 2,404x while maintaining sub-1MB total memory usage across platforms
• →User preference alignment improves RAG accuracy by 18.79 percentage points over baseline retrieval methods
• →Retrieval latency drops to 5-29ms per query, enabling practical real-time performance on consumer devices
• →The approach handles preference drift through streaming updates, accommodating evolving user preferences without full reindexing
• →Multi-benchmark validation across conversations, debates, and recommendations demonstrates generalization across diverse use cases