Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation

Researchers introduce ELF-S2T, a novel continuous-target generative model for speech-to-text tasks that combines audio conditioning with diffusion-based language modeling. The approach achieves competitive performance on ASR and speech translation while revealing that both tasks share common error patterns rooted in continuous latent space representations.

ELF-S2T represents a meaningful departure from discrete token generation in speech-to-text systems. Rather than generating individual text tokens sequentially, this model operates within a continuous latent space, leveraging pre-trained Embedded Language Flows (ELF) as its backbone. The architecture ingests speech through a frozen Whisper encoder and projects it into a continuous representation space, which then guides the denoising process during generation. This design choice enables a unified framework that handles both automatic speech recognition and speech translation tasks simultaneously.

The research builds on growing momentum around continuous-target language modeling, which has shown promise in other generative tasks. By conditioning a diffusion model on audio inputs, the researchers create a bridge between speech understanding and text generation without relying on intermediate discrete representations. The introduction of audio forcing during training and classifier-free guidance during inference demonstrates sophisticated techniques for ensuring the model properly leverages audio signals rather than defaulting to its pre-trained text priors.

The most compelling finding concerns error analysis: both ASR and speech translation errors stem from the same underlying mechanism—confusion in continuous latent space caused by nearby semantic representations. This discovery validates the continuous representation paradigm and suggests that recognition and translation share a common semantic mapping process. For developers building speech systems, this implies that improvements to latent space organization could simultaneously benefit multiple downstream tasks.

The public release of code and models accelerates adoption within the research community. As speech-to-text systems power real-world applications from accessibility tools to global communication platforms, advances in this domain have meaningful practical implications beyond academic interest.

• →ELF-S2T introduces continuous-target generation for speech-to-text tasks, moving away from traditional discrete token approaches
• →The model achieves competitive performance on both ASR and speech translation with a unified architecture
• →Error analysis reveals ASR and translation errors share common causes rooted in continuous latent space confusions
• →Audio forcing during training and classifier-free guidance at inference improve model reliance on speech signals
• →Public code release enables broader adoption and reproducibility within the speech AI research community