VITA-QinYu: Expressive Spoken Language Model for Role-Playing and Singing

Researchers unveiled VITA-QinYu, an expressive spoken language model that extends beyond natural conversation to generate role-playing and singing through a hybrid speech-text architecture. The model achieves state-of-the-art performance on conversational benchmarks while demonstrating superior expressiveness in non-conversational tasks, with researchers open-sourcing the code and providing a streaming-capable demo.

VITA-QinYu represents a meaningful advancement in spoken language model capabilities by addressing a significant gap in current SLM development. While existing models focus primarily on natural conversation, this system integrates paralinguistic elements—tone, emotion, performance style—into a unified framework. The hybrid speech-text paradigm with multi-codebook audio tokens allows the model to preserve modality separation while capturing richer expressive nuance, a technical approach that avoids the interference problems plaguing earlier attempts to merge linguistic and performance elements.

The development trajectory reflects broader industry recognition that human communication encompasses far more than semantic content. Traditional speech models treat expressiveness as secondary, but applications from entertainment to accessibility increasingly demand personality and emotional authenticity. VITA-QinYu's comprehensive training dataset of 15.8K hours—synthesized across conversational, role-playing, and singing domains—establishes a new benchmark for specialized speech model training infrastructure.

The competitive results are noteworthy: 7-point improvements on role-playing benchmarks and measurable gains on singing quality suggest the architecture successfully handles diverse expressive modalities without sacrificing conversational performance. The 1.38 and 4.98 percentage-point improvements on C3 and URO benchmarks indicate the model maintains baseline conversation quality while expanding capability scope.

The open-source release with streaming and full-duplex support significantly lowers barriers for developers integrating expressive speech into applications. This democratization accelerates adoption across gaming, virtual assistants, content creation, and entertainment sectors. Future development likely focuses on fine-tuning expressiveness across languages and integrating real-time emotional adaptation.

• →VITA-QinYu introduces multi-codebook audio tokens enabling richer paralinguistic representation while maintaining clear modality separation in spoken language models.
• →The model achieves 7-point improvements over peer SLMs on role-playing benchmarks while maintaining state-of-the-art conversational accuracy and fluency.
• →A 15.8K-hour synthesized training dataset spanning conversation, role-playing, and singing establishes new standards for expressive speech model development.
• →Open-source release with streaming and full-duplex support enables broader developer adoption for entertainment, accessibility, and virtual assistant applications.
• →The hybrid speech-text architecture successfully balances domain specialization in expressive tasks without degrading baseline conversational performance.