HoliTok:A Coutinuous Holistic Tokenization with Robust Dual Capabilities of Speech Generation and Understanding

HoliTok is a new continuous speech tokenization model that unifies speech generation and understanding tasks by encoding 48kHz audio into compact 128-dimensional latent sequences at 25Hz. The breakthrough addresses a key challenge in building unified speech foundation models by creating a tokenization space that balances reconstruction fidelity, semantic preservation, and learnability without requiring architectural workarounds.

HoliTok represents a significant advancement in unified speech modeling, solving a fundamental tension that has plagued the field. Previous speech tokenizers typically excelled at either high-fidelity reconstruction or semantic understanding, but struggled to serve both generation and recognition tasks simultaneously. This architectural fragmentation forced researchers to build separate pathways and optimization tricks into their models. HoliTok's continuous latent space—operating at 25Hz with 128 dimensions—achieves a sweet spot that preserves signal-level quality while maintaining learnability for language models.

The underlying innovation lies in the progressive training strategy that jointly optimizes three competing objectives: signal fidelity, semantic information, and latent learnability. This contrasts with existing approaches that sacrifice one dimension to maximize others. The unified AR+DiT architecture demonstrates that the same tokenization can power both generation-specific tasks like speech synthesis and joint generation-understanding tasks, eliminating the need for separate encoding schemes.

For the broader AI and speech foundation model ecosystem, HoliTok establishes a cleaner foundation for developing more efficient unified models. Developers can now focus on downstream modeling rather than designing custom tokenization schemes. The public code release accelerates adoption and standardization across the research community, potentially becoming a benchmark representation for spoken language modeling similar to how spectrograms served previous generations.

Looking forward, the success of continuous tokenization with dual capabilities suggests a trend toward simpler, more unified architectures in speech AI. The ability to operate robustly without additional optimization tricks positions HoliTok as a potential standard for future speech foundation models.

• →HoliTok introduces a continuous tokenization scheme encoding 48kHz speech into 25Hz sequences of 128-dimensional latents suitable for both generation and understanding tasks
• →Progressive training strategy simultaneously optimizes signal fidelity, semantic information, and language model learnability without architectural workarounds
• →Unified AR+DiT model achieves competitive reconstruction quality while improving generative learnability for speech synthesis and recognition
• →Public code availability accelerates standardization of speech tokenization across the research community
• →Addresses fundamental challenge of creating single tokenization space for unified speech foundation models