On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation

Researchers challenge the widespread practice of using global token perplexity to evaluate generative spoken language models, arguing this metric fails to account for fundamental differences between speech and text modalities. The study proposes alternative likelihood- and generative-based evaluation methods that correlate more strongly with human perception, revealing that performance gaps between leading models and human baselines are smaller than previously believed.

The evaluation of machine learning models fundamentally shapes how researchers prioritize development efforts and claim progress. This paper identifies a methodological blind spot in spoken language model assessment that has likely misdirected the field's understanding of actual model capabilities. By directly applying text-based perplexity metrics to speech tokens, the research community has been using a measurement tool designed for one modality on an entirely different one—analogous to assessing image quality using text comprehension standards.

Spoken language differs critically from written text in temporal continuity, acoustic variation, and emotional expression. Speech tokens encode these dimensions differently than text tokens, making direct perplexity comparison misleading. The authors demonstrate that their proposed metrics—which account for these modality-specific characteristics—show stronger correlations with human mean opinion scores, the gold standard for subjective quality assessment. This validation matters because it suggests previous benchmarking was systematically underestimating model quality.

For the AI development community, this research has immediate implications for resource allocation and investment priorities. If current evaluation metrics understate progress, teams may be overcorrecting through unnecessary optimization or infrastructure investment. The reshuffled performance landscape could alter which approaches receive funding and attention. More broadly, this work exemplifies how evaluation methodology directly influences technological progress narratives and competitive positioning within the field. Developers and researchers working on spoken language models should reassess their benchmarking approaches against these proposed metrics. The findings also highlight a broader principle: evaluation frameworks must be tailored to domain-specific characteristics rather than mechanically borrowed from adjacent fields.

• →Global token perplexity inappropriately applies text-based metrics to speech without accounting for modality-specific differences.
• →Proposed alternative metrics show stronger correlation with human perception (MOS) than traditional global perplexity.
• →Performance gaps between leading models and human baselines are significantly smaller than previously measured.
• →Evaluation methodology directly influences which models receive development resources and industry recognition.
• →Researchers should adopt modality-specific evaluation frameworks rather than mechanically transferring metrics across domains.