Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Researchers have developed techniques to enable fine-grained speaking style control in prompt-based text-to-speech models, allowing for smooth style transitions both between utterances and within single utterances. The approach uses embedding space interpolation for inter-utterance changes and attention mechanism modifications for intra-utterance style shifts, achieving high success rates in gender conversion and natural speaker transitions.

This research addresses a fundamental limitation in current prompt-based text-to-speech systems: their inability to perform granular, dynamic style control across time and content. While existing TTS models can generate natural speech from natural language prompts, they typically apply uniform speaking styles throughout an utterance, limiting their utility for applications requiring nuanced vocal expression that varies with narrative or emotional content.

The technical innovation centers on two complementary approaches. For smooth transitions between utterances, the researchers developed a method to compute direction vectors in the embedding space between different style prompts, enabling interpolation that produces gradual style shifts. More significantly, they identified and solved a critical architectural problem: autoregressive TTS decoders exhibit strong attention bias toward early tokens, causing initial audio characteristics to dominate the entire generation. By implementing KV-cache swapping and sliding-window attention masking, they reduced this bias and enabled genuine within-utterance style transitions.

The experimental results demonstrate practical viability across multiple dimensions. Gender conversion achieved 99-100% success rates, pitch variations reached 36 Hz, and speech rate changes reached 1.6 syllables per second—all metrics relevant to real-world voice synthesis applications. The maintenance of speaker similarity scores between 0.81-0.91 during transitions indicates the system preserves voice identity while modulating style, critical for professional audio applications.

This advancement has implications for audiobook production, podcast creation, synthetic media generation, and accessibility applications. The ability to programmatically control fine-grained vocal qualities opens possibilities for more expressive and contextually appropriate synthetic speech across diverse use cases.

• →Embedding space interpolation enables smooth speaking style transitions across utterances with up to 36 Hz pitch variation capability
• →Attention mechanism modifications solve the early-token bias problem in autoregressive TTS decoders, enabling intra-utterance style changes
• →Gender conversion achieves 99-100% success rate while maintaining speaker similarity of 0.81-0.91 during transitions
• →Proposed techniques work within existing prompt-based TTS models without requiring architectural redesign
• →Technology enables practical applications in audiobooks, podcasts, and contextually-aware synthetic speech generation