40 articles tagged with #speech-synthesis. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose a fully end-to-end training framework that jointly optimizes all components of discrete-token-based text-to-speech systems—speech tokenizers, language models, diffusion models, and reward models—rather than training them independently. The approach achieves state-of-the-art results on benchmark tests with smaller, more efficient models.
Researchers introduce Audio-FLAN, a large-scale instruction-tuning dataset with over 100 million instances covering 80 diverse tasks across speech, music, and sound domains. This dataset addresses a critical gap in unified audio-language models by enabling both audio understanding and generation tasks, advancing the integration of audio capabilities into large language models.
Researchers introduce TLDR, a patch-based autoregressive framework that compresses audio tokens to accelerate text-to-speech synthesis. The method achieves 1.8x inference speedup and reduces KV-cache memory by 75% without replacing existing model modules, addressing a key efficiency bottleneck in codec-based speech language models.
Researchers have developed dots.tts, a 2-billion parameter text-to-speech model that achieves state-of-the-art performance through innovations in continuous speech modeling, full-history conditioning, and self-corrective training. The model demonstrates exceptional multilingual capabilities and enables low-latency speech generation, with code and weights released open-source under Apache 2.0 license.
HoliTok is a new continuous speech tokenization model that unifies speech generation and understanding tasks by encoding 48kHz audio into compact 128-dimensional latent sequences at 25Hz. The breakthrough addresses a key challenge in building unified speech foundation models by creating a tokenization space that balances reconstruction fidelity, semantic preservation, and learnability without requiring architectural workarounds.
Researchers address a critical limitation in Spoken Language Models (SLMs) for low-resource languages by identifying a fundamental trade-off called the Stability-Expressivity Gap, where synthetic data improves phonetic accuracy but suppresses prosodic variability. The proposed self-alignment frameworks—DGSA and TDSC—recover expressivity while maintaining stability, achieving performance comparable to commercial systems and enabling zero-shot voice cloning for Lao.
PilotTTS demonstrates that competitive text-to-speech systems no longer require massive proprietary datasets or complex architectures. Using only 200K hours of openly-processed data and a lightweight autoregressive model, the system achieves industry-leading performance on benchmark tests while supporting voice cloning, emotion synthesis, and multilingual capabilities.
X-Voice is a 0.4B multilingual voice cloning model that enables zero-shot cross-lingual speech synthesis across 30 languages using a two-stage training approach with IPA as a unified representation. The open-sourced system achieves performance comparable to billion-scale models while eliminating the need for transcribed audio prompts, advancing accessibility in multilingual AI-generated speech.
Researchers introduce CrossAccent-TTS, a machine learning framework that enables precise control over accent characteristics in cross-lingual text-to-speech systems. The technology uses an Accent Intensity Controller to allow smooth interpolation between accents while maintaining speaker identity, with particular applications for low-resource Indic languages.
Researchers introduce Sarashina2.2-TTS, a Japanese-focused text-to-speech system trained on 361k hours of speech that addresses kanji polyphony challenges through scaled training and targeted data augmentation. The system achieves state-of-the-art performance on Japanese pronunciation while maintaining cross-lingual robustness, alongside a new benchmark for evaluating kanji reading accuracy.
Researchers introduce OscillaTTS, a diffusion-based text-to-speech system that uses adaptive oscillatory nonlinearity to better model sharp prosodic transitions and rapid pitch variations in expressive speech. The approach improves upon existing methods that rely on fixed periodic activation functions, demonstrating consistent improvements in both objective metrics and subjective evaluations on standard speech datasets.
Researchers have developed a District Guided Tokens (DGT) technique to improve Bengali text-to-IPA transcription by incorporating regional dialect information, with the ByT5 model achieving superior performance on a new dataset spanning six Bangladeshi districts. This advancement addresses the phonological complexity of Bengali dialects and demonstrates the importance of regional context in natural language processing systems.
Researchers propose LLM-Based Multi-Reference Evaluation (LMRE), a new method for assessing phrase break annotations in speech that acknowledges multiple valid phrasings rather than assuming a single correct interpretation. Tested on 1,356 Korean annotations, LMRE demonstrates stronger alignment with human judgment than traditional single-reference approaches, suggesting large language models can effectively evaluate prosodic speech characteristics at scale.
Bagpiper-TTS is a universal speech synthesis system that uses natural language prompts to guide flexible speech generation, moving beyond rigid TTS frameworks. The model achieves competitive performance across multiple applications including multi-talker synthesis, singing voice synthesis, and intent-to-speech tasks, matching dedicated models while offering broader versatility.
EmoInstruct-TTS introduces a dual-path framework for emotional speech synthesis that enables fine-grained emotional control through natural language instructions. The system uses Emotion2embed, covering 48 emotional states, and an Instruction-Conditioned Emotion Flow Model to convert free-form text instructions into acoustically grounded emotion representations integrated with LLM-based synthesis pipelines.
Researchers propose a cross-attention attribution method for style-captioned text-to-speech systems, adapting the DAAM framework to speech diffusion models for the first time. Analysis of 3,600 style-caption and text combinations reveals how individual words influence acoustic output, showing that style tokens condition voice characteristics globally while peaking in early generation steps and deep network layers.
Researchers demonstrate a method to repurpose pre-trained speech classifiers for conditional speech generation by attaching a lightweight subnetwork, eliminating the need for separate classifier and diffusion models. This approach reduces memory footprint and computational cost while maintaining high speech quality, bridging discriminative and generative modeling in a single unified architecture.
Researchers propose a feature-aligned speech watermarking method that embeds imperceptible identifiable information into audio while maintaining robustness against speech reconstruction models. By aligning watermarks with original speech feature distributions, the technique overcomes the traditional robustness-fidelity trade-off that has limited previous audio watermarking approaches.
Researchers introduce Whisper-GPT, a hybrid language model that combines continuous audio representations (spectrograms) with discrete acoustic tokens to improve speech and music generation. This approach addresses context length limitations in traditional token-based models while maintaining high-fidelity audio synthesis capabilities.
Researchers analyze how discrete speech units derived from self-supervised learning entangle phonetic, speaker, and language information in multilingual vocoder systems. The study demonstrates that cluster size directly controls intelligibility while explicit speaker conditioning prevents identity collapse, with implications for improving Audio LLMs and speech generation systems.
Researchers analyzed how voice cloning technology preserves accented speech compared to standard speech, finding that clones of accented speakers show larger perceptual differences from originals despite similar baseline-normalized embedding distances. The study reveals that accent variation significantly impacts perceived speaker identity and intelligibility in voice cloning systems, suggesting current speaker-discriminative embeddings don't fully capture accent preservation.
Researchers introduce Chatterbox-Flash, a zero-shot text-to-speech model combining block-diffusion decoding with streaming capabilities. The system addresses token distribution bias through prior-calibrated scoring and early-decoding schedules, achieving high-fidelity speech synthesis with low latency comparable to autoregressive systems.
Researchers introduce ImmersiveTTS, an AI model that generates natural speech integrated within environmental audio contexts using multimodal diffusion transformers and domain-specific representation alignment. The advancement addresses a key challenge in audio generation: seamlessly combining speech with background environmental sounds while maintaining acoustic quality and intelligibility.
Alibaba's Fun-Realtime-TTS-Preview voice AI model ranked fifth on the Artificial Analysis Speech Arena leaderboard, outperforming systems from OpenAI and xAI. This achievement marks Alibaba as the only Chinese-engineered voice system in the global top five, supporting 30+ languages and multiple Chinese dialects.
Researchers introduce LoSATok, a novel audio tokenizer that compresses high-dimensional semantic features into 128-dimensional representations while preserving understanding and generation capabilities. The innovation combines semantic bottleneck compression with dual-level supervision to improve performance for speech, music, and audio generation tasks across diffusion transformer models.
