KARMA-MV: A Benchmark for Causal Question Answering on Music Videos

Researchers introduce KARMA-MV, a large-scale dataset of 37,737 multiple-choice questions derived from 2,682 YouTube music videos, designed to benchmark AI models' ability to reason about causal relationships between visual dynamics and musical structure. The dataset leverages LLM-based generation for scalability and proposes a causal knowledge graph approach to improve vision-language model performance on cross-modal audio-visual reasoning tasks.

KARMA-MV addresses a significant gap in multimodal AI research by focusing on causal reasoning in music videos—a domain requiring integration of temporal audio-visual cues that traditional datasets have largely ignored. The benchmark moves beyond correlation-based understanding to test how visual elements causally influence musical choices, a more sophisticated reasoning task. The dataset's innovative use of LLM-driven generation rather than manual annotation enables unprecedented scale while maintaining quality through validation mechanisms.

This research emerges from rapid advances in vision-language models and cross-modal understanding, yet most existing benchmarks emphasize passive description or simple retrieval rather than causal inference. Music videos present a unique testing ground because they inherently encode intentional relationships between visual and audio elements, making causal reasoning verifiable and meaningful for downstream applications in creative AI tools, content generation, and multimedia understanding.

The causal knowledge graph approach demonstrates that augmenting VLMs with structured cross-modal dependencies consistently improves performance, with particularly pronounced gains for smaller models. This finding has implications for resource-constrained deployments and suggests explicit causal structure is critical for complex reasoning. The work establishes a framework applicable beyond music videos—to film, advertising, and interactive media where visual-audio synchronization carries semantic weight.

Future research will likely extend this methodology to other domains requiring tight audio-visual integration. The benchmark may accelerate development of AI systems capable of understanding and generating synchronized multimedia content, with applications in entertainment production, accessibility tools, and creative assistance platforms.

• →KARMA-MV provides 37,737 causal reasoning questions across 2,682 music videos, enabling benchmarking of audio-visual understanding beyond correlation
• →LLM-based dataset generation achieves scalability while maintaining quality, offering a template for future multimodal benchmark construction
• →Causal knowledge graphs significantly improve VLM performance on cross-modal reasoning, particularly benefiting smaller models with limited parameters
• →The dataset focuses on causal inference rather than passive description, testing models' ability to understand why visual elements drive musical choices
• →Results suggest explicit causal structure is essential for multimodal reasoning tasks that require understanding intentional design relationships