AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding

Researchers introduce AMUSE, a new benchmark for evaluating multimodal large language models in multi-speaker dialogue scenarios. The framework addresses current limitations of models like GPT-4o in tracking speakers, maintaining conversational roles, and reasoning across audio-visual streams in applications such as conversational video assistants.

• →Current multimodal AI models struggle with complex multi-speaker dialogue understanding despite strong general perception capabilities.
• →AMUSE benchmark focuses on agentic reasoning tasks requiring speaker tracking and role maintenance across time.
• →The framework targets applications in conversational video assistants and meeting analytics.
• →Models must jointly reason over both audio and visual streams simultaneously.
• →This addresses a critical gap in multimodal AI evaluation for real-world conversational scenarios.