WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs

Researchers have introduced WorldSense, the first benchmark for evaluating multimodal AI systems that process visual, audio, and text inputs simultaneously. The benchmark contains 1,662 synchronized audio-visual videos across 67 subcategories and 3,172 QA pairs, revealing that current state-of-the-art models achieve only 65.1% accuracy on real-world understanding tasks.

• →WorldSense is the first comprehensive benchmark to evaluate AI models on combined visual, audio, and text understanding.
• →The benchmark includes 1,662 audio-visual videos categorized into 8 domains and 67 subcategories with expert-annotated QA pairs.
• →Current state-of-the-art multimodal AI models achieve only 65.1% accuracy, indicating significant room for improvement.
• →The benchmark requires strong coupling between audio and video inputs, testing true omnimodal perception capabilities.
• →All annotations were manually created by 80 expert annotators with multiple quality control rounds.