M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Researchers introduce M³Eval, the first comprehensive benchmark for evaluating memory capabilities in multi-modal AI models processing long-form video. Testing across multiple models reveals significant weaknesses in maintaining disentangled representations, handling temporal information, and symbolic memory—highlighting memory as a critical yet understudied dimension of AI development.

M³Eval addresses a notable gap in AI evaluation frameworks by systematically assessing memory—a capability increasingly important as models handle longer, more complex video sequences. While existing benchmarks focus on perception and reasoning tasks, memory evaluation has remained fragmented and ad-hoc. This research, grounded in cognitive psychology principles, isolates specific memory dimensions through carefully designed tasks, providing the first rigorous methodology for comparing how different models retain and process information.

The findings reveal substantial performance gaps between current models and human-like memory. Models struggle when processing parallel video streams simultaneously, suggesting architectural limitations in maintaining separate information contexts. Critically, interference patterns in AI systems differ markedly from human cognition, implying models use fundamentally different memory mechanisms than biological systems. The framework also identifies that spatial grounding outperforms temporal grounding—models better remember where objects appear than when events occur—pointing to specific areas requiring architectural innovation.

For the AI development community, M³Eval provides both diagnostic value and practical direction. Developers can now benchmark progress on memory-specific capabilities rather than relying on general performance metrics. This targeted evaluation accelerates the path toward models with more robust, human-aligned memory systems. The open-source release of code and datasets democratizes access to these evaluations, likely spurring competitive improvements across model architectures.

Looking forward, memory evaluation will likely become standard in multi-modal model assessment, similar to existing benchmarks for vision or language tasks. Teams will compete on memory robustness metrics, driving innovation in attention mechanisms, context windows, and temporal reasoning—capabilities essential for real-world applications in video understanding, autonomous systems, and long-context analysis.

• →M³Eval is the first comprehensive benchmark systematically evaluating memory capabilities in multi-modal models processing video.
• →Current models struggle to maintain separate representations when processing parallel video streams and show poor temporal grounding compared to spatial grounding.
• →AI interference patterns differ substantially from human memory, suggesting fundamentally different underlying mechanisms.
• →Limited symbolic memory capacity emerged as a distinctive weakness across tested multi-modal models.
• →Open-source availability of M³Eval will likely establish memory evaluation as a standard component in future model benchmarking.