SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding

Researchers introduce SONIC-O1, a comprehensive benchmark for evaluating multimodal large language models on audio-video understanding tasks. The study reveals significant performance gaps between closed-source and open-source models, particularly in temporal localization, and identifies demographic disparities in model behavior across 60 hours of real-world conversational data.

SONIC-O1 addresses a critical gap in AI evaluation methodologies by focusing on sequential audio-video understanding rather than static image analysis. As multimodal AI models become increasingly prevalent, the lack of standardized benchmarks for temporal reasoning has hindered systematic performance assessment. This benchmark enables researchers to measure three distinct capabilities: open-ended summarization, multiple-choice question answering, and temporal localization with reasoning—each representing different cognitive demands that real-world applications require.

The research reveals meaningful performance disparities that warrant attention from developers and researchers. While multiple-choice accuracy remains relatively consistent across model families, closed-source models demonstrate a 22.6% advantage over open-source alternatives in temporal localization—a task requiring understanding of sequential relationships within video content. More significantly, the study documents accuracy variations of up to 21.4% across demographic groups, suggesting that current multimodal models may exhibit systematic biases in how they process and understand temporal information across different user populations.

These findings carry implications for AI practitioners developing real-world applications. Companies deploying multimodal models for video analysis, content understanding, or accessibility features must account for performance variability across both model architectures and user demographics. The public availability of SONIC-O1 through HuggingFace, GitHub, and a dedicated leaderboard democratizes access to rigorous evaluation standards, potentially accelerating efforts to develop more robust and equitable multimodal systems. Organizations should monitor leaderboard developments to track progress on temporal grounding tasks.

• →SONIC-O1 provides the first comprehensive benchmark for evaluating multimodal LLMs on real-world audio-video understanding across 13 conversational domains.
• →Closed-source models outperform open-source models by 22.6% on temporal localization tasks, indicating significant capability gaps.
• →Demographic disparities of up to 21.4% in temporal localization accuracy reveal persistent bias issues in multimodal AI systems.
• →The benchmark evaluates three distinct capabilities: summarization, multiple-choice answering, and temporally-grounded reasoning.
• →Public availability of the dataset and leaderboard enables ongoing community-driven improvements to multimodal AI evaluation standards.