From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

Researchers have developed MM-Mem, a new pyramidal multimodal memory architecture that enables AI systems to better understand long-horizon videos by mimicking human cognitive memory processes. The system addresses current limitations in multimodal large language models by creating a hierarchical memory structure that progressively distills detailed visual information into high-level semantic understanding.

• →MM-Mem introduces a three-tier memory architecture (Sensory Buffer, Episodic Stream, Symbolic Schema) inspired by cognitive science theory.
• →The system addresses key limitations of existing approaches that either suffer from high latency or lose important details through aggressive compression.
• →A Semantic Information Bottleneck objective with SIB-GRPO optimization balances memory compression with task-relevant information retention.
• →An entropy-driven retrieval strategy allows the system to access memory hierarchically, starting with abstract concepts and drilling down when needed.
• →Extensive testing across 4 benchmarks demonstrates effectiveness for both offline and streaming video analysis tasks.