KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs

Researchers introduce three new Korean speech benchmarks (KVoiceBench, KOpenAudioBench, and KMMAU) totaling 12,345 samples to evaluate multilingual speech language models, addressing the gap in non-English evaluation. The study reveals significant performance disparities between English and Korean across eight SpeechLMs, exposing weaknesses invisible to English-only testing.

The development of language-specific speech benchmarks addresses a critical gap in AI evaluation infrastructure. While speech language models have advanced substantially by extending LLMs to audio, their assessment remains dominated by English-language tasks, creating blind spots for multilingual capabilities. This research demonstrates that simple benchmark translation through ASR and TTS pipelines corrupts language-specific properties, making native benchmarks essential for accurate evaluation.

The three Korean benchmarks represent a methodological shift in how researchers should approach multilingual AI evaluation. Rather than retrofitting English benchmarks, the authors developed human-agent frameworks that preserve linguistic nuances, speaker attributes, and paralinguistic properties specific to Korean. This approach recognizes that audio understanding requires preserving accent, tone, and cultural context—elements lost in mechanical translation.

The evaluation findings carry significant implications for SpeechLM developers and users. The divergence between English-Korean performance rankings and the variance across task families suggests that models optimized for English may fail unpredictably in other languages. This fragmentation creates reliability concerns for deploying multilingual speech systems in production environments. Companies targeting Korean markets cannot confidently extrapolate English benchmark performance.

Looking forward, this work establishes a template for benchmark construction in other languages and demonstrates the necessity of comprehensive multilingual evaluation suites. As speech models expand globally, the industry requires similar frameworks for Mandarin, Hindi, Japanese, and other major languages. The public release of these benchmarks enables ongoing research but also highlights how far the field remains from truly language-agnostic speech understanding.

• →Korean speech benchmarks reveal substantial English-Korean performance gaps across eight SpeechLMs, indicating language-specific model weaknesses.
• →Direct translation of English benchmarks corrupts language-specific instructions and audio properties, necessitating native-language benchmark construction.
• →SpokenQA and audio understanding task rankings diverge significantly, exposing complementary model weaknesses invisible to single-task evaluation.
• →The 12,345-sample Korean benchmark suite (three components) establishes a replicable methodology for multilingual AI evaluation.
• →Speech model deployment in non-English markets requires language-specific evaluation to ensure reliability and performance consistency.