Interpreting and Steering a Text-to-Speech Language Model with Sparse Autoencoders

Researchers have developed sparse autoencoders to interpret and control how language models process text-to-speech synthesis in CosyVoice3. The work demonstrates that interpretable features—phonemes, laughter, accent, and speaker gender—are causally linked to speech output and can be precisely steered to modify synthesis behavior without retraining.

This research addresses a critical gap in understanding how modern language models handle multimodal tasks where text and speech tokens coexist in the same computational space. By applying BatchTopK sparse autoencoders to CosyVoice3's LM backbone, researchers created a systematic method to reverse-engineer the model's internal representations, moving beyond black-box analysis toward mechanistic interpretability.

The work builds on growing recognition that sparse autoencoders can decompose complex neural representations into human-interpretable features. Unlike prior interpretability research that merely identifies what models do, this study proves causality through targeted interventions—flipping speaker gender, increasing laughter probability 40-fold, or modulating speech rate while preserving semantic content. This distinction matters because it shows these features are not epiphenomenal but functionally crucial to model behavior.

For AI safety and development communities, this demonstrates that TTS systems aren't inscrutable black boxes. The ability to surgically modify outputs through latent space steering has immediate applications: developers can control synthesis characteristics without retraining, while researchers gain tools to audit model behavior for bias or unwanted patterns. This interpretability approach is scalable—the methodology could extend to other multimodal architectures combining discrete and continuous modalities.

Looking forward, sparse autoencoders may become standard interpretability infrastructure for language models as they expand beyond text. The research signals momentum toward more transparent AI systems where developers understand not just what models output, but why and how to steer them purposefully.

• →Sparse autoencoders successfully decoded interpretable features in a text-to-speech language model, including phonemes, laughter, accent, and speaker gender.
• →Targeted interventions proved these features are causally linked to output rather than merely correlational, with dramatic effects like increasing laughter probability from 2% to 79%.
• →The methodology enables precise control of TTS synthesis through latent space steering without requiring model retraining.
• →This interpretability approach addresses AI safety concerns by making multimodal language model behavior more transparent and auditable.
• →The sparse autoencoder framework may become a standard tool for understanding and controlling large language models across different modalities.