IRAF: Interference-Resilient Adaptive Fusion for Noise-Robust End-to-End Full-Duplex Spoken Dialogue Systems

Researchers propose IRAF, a lightweight module that improves full-duplex spoken dialogue systems by filtering interference from background speakers. The technology uses adaptive fusion to modulate user audio reliability frame-by-frame, demonstrating improved response quality and stable turn-taking in noisy acoustic environments.

Full-duplex conversational AI represents a significant frontier in voice-based human-computer interaction, enabling agents to respond naturally with overlapping speech rather than waiting for users to finish. However, this capability introduces a critical technical challenge: background speaker interference corrupting the user's microphone stream degrades model performance and creates unstable interactions. IRAF addresses this vulnerability through a streaming-compatible module that learns to assess audio reliability in real-time, dynamically adjusting how much weight the language model assigns to potentially corrupted user audio segments.

This work emerges from broader efforts to make end-to-end dialogue systems more robust in real-world conditions. Traditional approaches either rely on explicit speaker diarization or accept degraded performance; IRAF offers a middle path by embedding reliability estimation directly into the fusion mechanism. The module predicts scalar gates from audio embeddings, operating efficiently without introducing significant latency—critical for conversational naturalness.

For developers building commercial voice assistants, IRAF's practical benefits are substantial. Testing on MS-MARCO and InstructS2S-200K datasets shows consistent quality improvements under interference, directly translating to fewer failed interactions and better user satisfaction. The lightweight design suggests deployment feasibility across various hardware platforms.

The research points toward more resilient multimodal systems where components gracefully degrade when signal quality degrades rather than failing catastrophically. Future work likely involves extending similar adaptive fusion principles to other modalities and exploring tighter integration with voice activity detection and echo cancellation pipelines.

• →IRAF uses frame-by-frame reliability gating to filter speaker interference from user microphone streams in full-duplex dialogue systems.
• →The module operates as a lightweight, streaming-compatible layer compatible with end-to-end LLM-based voice agents.
• →Testing demonstrates consistent improvements in response quality and turn-taking stability under realistic acoustic interference.
• →Adaptive fusion mechanisms represent a practical approach to robustness that avoids explicit speaker diarization overhead.
• →The technology addresses a key deployment challenge for real-world conversational AI systems in non-ideal acoustic environments.