OmniMem: Perturbation-aware Memory Compression for Streaming Audio-Visual LLMs

OmniMem is a new memory compression framework for audio-visual large language models that enables efficient long-form video understanding by using modality-aware memory allocation and perturbation-aware token selection. The approach achieves 2-4% accuracy improvements over existing compression methods while reducing memory requirements, with potential applications in real-time video AI systems.

OmniMem addresses a critical bottleneck in modern multimodal AI systems: the exponential memory growth that occurs when processing extended video content. As audio-visual LLMs become increasingly capable, their practical deployment remains constrained by the quadratic scaling of key-value caches during inference. This framework's innovation lies in recognizing that visual and audio streams present fundamentally different compression challenges due to their vastly different token densities, yet existing methods treat all tokens identically.

The research builds on years of work in model compression and efficient transformers, but applies these techniques with modality-specific intelligence. By implementing perturbation-aware selection—measuring which KV states genuinely contribute to model predictions—OmniMem preserves semantic understanding while aggressively pruning redundant information. The addition of budget-aware fine-tuning suggests the authors recognized that pre-trained models don't inherently optimize for compressed representations, requiring explicit training to consolidate information efficiently.

For the AI development community, this work has immediate practical implications. Video understanding remains a dominant use case for multimodal systems, from surveillance to content analysis to autonomous systems. Current memory constraints limit these applications to either short clips or heavily subsampled video, creating real deployment friction. The 2-4% absolute accuracy gains demonstrated on established benchmarks (VideoMME Long, LVBench, LVOmniBench) suggest the method maintains competitive performance while enabling longer inference sequences.

Future developments will likely focus on extending these compression principles to other modality combinations and exploring whether the techniques generalize across different model architectures. The work's emphasis on streaming inference suggests practical deployment scenarios where real-time processing matters, particularly in edge computing environments where memory budgets remain severely constrained.

• →OmniMem achieves 2-4% accuracy improvements by separately managing visual and audio token compression rather than treating all tokens uniformly.
• →Perturbation-aware memory selection identifies and preserves only the KV states that genuinely impact model predictions, enabling aggressive compression without sacrificing understanding.
• →Budget-aware fine-tuning adds 1-2% additional accuracy by training models to consolidate useful information into retained memory tokens.
• →The framework enables streaming inference on long-form video content, addressing the primary constraint limiting current audio-visual LLM deployment.
• →Experiments on video-SALMONN 2+ and Qwen-2.5-Omni demonstrate consistent improvements across multiple benchmarks, suggesting broad applicability.