HumanVBench: Probing Human-Centric Video Understanding in MLLMs with Automatically Synthesized Benchmarks

Researchers introduced HumanVBench, a comprehensive benchmark for evaluating how well multimodal AI models understand human-centric video content across 16 tasks including emotion recognition and speech-visual alignment. The study evaluated 30 leading MLLMs and found significant performance gaps, even among top proprietary models, while introducing automated synthesis pipelines to enable scalable benchmark creation with minimal human effort.

HumanVBench addresses a critical evaluation gap in multimodal AI development by systematizing the assessment of human-centric understanding—a capability increasingly central to real-world applications. The benchmark's novel automated synthesis approach, which converts model errors into plausible distractors, provides researchers a replicable framework for generating nuanced evaluation datasets without prohibitive annotation costs. This methodological innovation matters because high-quality benchmarking has historically been a bottleneck limiting rapid AI progress; automating this process could accelerate development cycles across the industry.

The research exposes meaningful deficiencies in current MLLMs, particularly in subtle emotion perception and cross-modal synchronization tasks where even leading proprietary models underperform humans. This finding is contextually important given the increasing deployment of vision-language models in content moderation, accessibility tools, and social intelligence applications where these capabilities directly impact user experience and safety. The 16 fine-grained task structure provides developers granular insights into specific capability gaps rather than aggregate scores.

For the AI development community, HumanVBench functions as both a diagnostic tool and a standardized reference point for measuring progress. Open-sourcing the benchmark and synthesis pipelines democratizes access to rigorous evaluation infrastructure, potentially raising baseline standards across smaller research groups and organizations. This competitive pressure may drive focused improvements in human-centric understanding, moving the field beyond generic video captioning toward socially-aware systems.

• →HumanVBench introduces automated synthesis pipelines that generate high-quality video benchmarks with minimal human annotation, addressing scalability bottlenecks in AI evaluation
• →Evaluation of 30 leading MLLMs reveals critical gaps in emotion recognition and speech-visual alignment, even among top proprietary models
• →The benchmark's 16 fine-grained tasks provide granular diagnostic capabilities for identifying specific performance deficiencies in video understanding models
• →Open-sourcing the methodology enables broader access to sophisticated benchmark construction techniques across research organizations
• →Results suggest human-centric video understanding remains a significant frontier for MLLM development, with implications for accessibility and content moderation applications