DSA-Tokenizer: Disentangled Semantic-Acoustic Tokenization via Flow Matching-based Hierarchical Fusion

Researchers introduce DSA-Tokenizer, a novel speech tokenization system that separates semantic content from acoustic style using distinct optimization paths and Flow Matching decoders. The approach enables discrete Speech LLMs to achieve better disentanglement while supporting efficient voice cloning and high-fidelity speech generation with minimal inference steps.

DSA-Tokenizer addresses a fundamental challenge in speech processing: how to cleanly separate linguistic meaning from speaker characteristics and acoustic properties. Traditional tokenizers struggle with this separation, either prioritizing one aspect over another or failing to achieve true disentanglement. This research tackles the problem through dual optimization constraints where semantic tokens learn linguistic content via ASR supervision while acoustic tokens focus on reconstructing mel-spectrograms to capture style information. The introduction of a hierarchical Flow Matching decoder represents a technical advancement in generative modeling for speech, enabling both reconstruction and cross-utterance voice cloning capabilities.

The significance extends beyond academic novelty. Speech LLMs require discrete token representations to function efficiently, and better semantic-acoustic separation directly improves downstream model performance and controllability. The paper's distillation strategy reduces inference sampling steps to just four iterations while maintaining synthesis quality through GAN fine-tuning, addressing practical deployment concerns. This efficiency gain matters for real-time applications and resource-constrained environments. The joint reconstruction-context inpainting training strategy demonstrates how innovative training methodologies can unlock new capabilities in tokenization systems.

For the broader AI infrastructure landscape, this work provides a more effective interface for large-model speech generation. Developers building conversational AI, voice assistants, or speech synthesis applications could benefit from cleaner semantic-acoustic disentanglement, enabling better control over voice characteristics while preserving linguistic accuracy. The research suggests that properly designed tokenization layers serve as crucial foundations for downstream model performance, influencing how future speech-based AI systems are architected and optimized.

• →DSA-Tokenizer achieves explicit semantic-acoustic disentanglement through distinct optimization constraints supervised by ASR and mel-spectrogram reconstruction.
• →Hierarchical Flow Matching decoder with distillation reduces inference to 4 sampling steps while improving synthesis quality via GAN fine-tuning.
• →Joint reconstruction and context inpainting training enables both high-fidelity generation and cross-utterance voice cloning capabilities.
• →Disentangled tokenization provides a more effective interface for downstream large-model speech generation with improved WER/CER metrics.
• →The approach addresses practical deployment concerns by achieving efficient inference without sacrificing audio quality or controllability.