GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models

GlobeAudio, a new benchmark dataset, evaluates Large Audio-Language Models across six languages using 5,637 naturally-sourced audio questions. The research reveals significant performance gaps in current LALMs, particularly for open-source models and low-resource languages, highlighting critical limitations in how audio-language AI systems handle real-world acoustic conditions.

The development of GlobeAudio addresses a fundamental gap in AI evaluation methodology. While Large Audio-Language Models have proliferated rapidly, standardized benchmarks for assessing their real-world performance remain sparse and often rely on synthetic or cleaned audio data rather than naturalistic conditions. This research bridges that divide by introducing a dataset grounded in authentic linguistic and cultural contexts, created by native speakers across typologically diverse languages.

The significance extends beyond academic rigor. Current LALM evaluation frameworks typically ignore acoustic realism—background noise, accents, speech variations—that characterize actual deployment scenarios. LALMs power critical applications from accessibility tools to multilingual voice interfaces, where performance degradation directly impacts user experience. The discovery that open-source models and low-resource language implementations show substantial performance gaps under natural conditions suggests the field has optimized primarily for benchmark scores rather than practical utility.

For developers and researchers, GlobeAudio forces a reckoning with evaluation standards. The benchmark's emphasis on higher-level auditory reasoning and cultural grounding exposes how current models may excel at pattern matching while failing at nuanced comprehension. This has immediate implications for deployment decisions, as organizations cannot reliably assume performance on clean test data translates to production environments.

Looking forward, this research will likely catalyze more rigorous evaluation practices across the LALM landscape. The public release through Hugging Face ensures broad adoption, positioning GlobeAudio as a standard reference point. Future model development will face pressure to demonstrate performance across naturalistic conditions and underrepresented languages, potentially reshaping architectural priorities in the audio-language modeling community.

• →GlobeAudio contains 5,637 expert-crafted questions across six languages, emphasizing naturalistic audio and cultural authenticity.
• →Current open-source LALMs show significant performance degradation under natural acoustic conditions compared to closed-source alternatives.
• →Low-resource languages exhibit the largest performance gaps, indicating unequal progress in multilingual audio-language modeling.
• →Most existing LALM evaluations fail to capture real-world acoustic realism, potentially inflating perceived model capabilities.
• →The benchmark is publicly available on Hugging Face, likely to become a standard evaluation reference for the field.