Multilingual Multi-Speaker Unit Vocoders: A Systematic Analysis of Discrete Speech Representations

Researchers analyze how discrete speech units derived from self-supervised learning entangle phonetic, speaker, and language information in multilingual vocoder systems. The study demonstrates that cluster size directly controls intelligibility while explicit speaker conditioning prevents identity collapse, with implications for improving Audio LLMs and speech generation systems.

This research addresses a fundamental limitation in speech generation technology that has received minimal scrutiny despite widespread deployment in audio language models and cross-lingual speech systems. Discrete speech units—created by clustering self-supervised embeddings—compress complex acoustic information but inadvertently mix speaker identity, phonetic content, and language characteristics, leading to degraded output quality in multilingual contexts.

The systematic analysis of BigVGAN-based vocoders across four Indian languages reveals critical design trade-offs. Cluster size emergence as the primary lever for phonetic discriminability shows that larger inventories better separate similar phonemes across different languages, while smaller clusters cause cross-lingual phoneme collapse. This finding contradicts assumptions that bigger is always better, instead indicating an optimal balance point dependent on language characteristics and downstream task requirements.

The necessity of explicit speaker conditioning signals a deeper architectural limitation: without dedicated identity controls, neural vocoders naturally collapse speaker variation into the discrete unit space itself. This has immediate implications for developers building multilingual voice systems, requiring explicit architectural decisions beyond basic clustering approaches. Language supervision adds incremental gains primarily when phonetic ambiguity remains high, suggesting diminishing returns in well-separated phonetic spaces.

For Audio LLM developers and speech synthesis companies, these findings indicate that vocoder design directly impacts model capability ceiling. The research suggests that next-generation systems should incorporate adaptive clustering strategies and mandatory speaker/language conditioning layers rather than treating vocoders as interchangeable components. Organizations deploying multilingual speech systems should validate these relationships empirically for their specific language pairs before production deployment.

• →Cluster size directly governs speech intelligibility by controlling phonetic discriminability across languages
• →Explicit speaker conditioning is architecturally essential to prevent speaker identity collapse in multilingual contexts
• →Similar phonemes across languages collapse into identical clusters at smaller inventories, progressively separating with larger cluster sizes
• →Language supervision provides greatest gains at lower cluster sizes where phonetic ambiguity remains high
• →Unit vocoder design directly impacts Audio LLM capability ceilings and requires language-specific empirical validation