MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models

Researchers introduce MTR-DuplexBench, a new evaluation framework for Full-Duplex Speech Language Models that enables real-time overlapping conversations. The benchmark addresses critical gaps by assessing multi-round interactions across conversational quality, instruction-following, and safety dimensions, revealing that current FD-SLMs struggle with consistency across multiple communication rounds.

The introduction of MTR-DuplexBench represents a significant advancement in AI model evaluation methodology. Full-duplex speech language models represent the next generation of conversational AI, enabling natural overlapping dialogue similar to human conversation rather than the rigid turn-taking of traditional systems. However, the field has lacked robust evaluation frameworks for assessing these models in realistic multi-round scenarios where conversation complexity compounds across multiple exchanges.

This benchmark addresses a genuine technical gap that has emerged as FD-SLMs move from research prototypes toward practical deployment. Previous evaluation methods focused narrowly on single-turn interactions and conversational metrics alone, missing critical performance dimensions like instruction adherence and safety guardrails during extended dialogues. The challenge of segmenting continuous full-duplex speech into discrete evaluable turns, while maintaining context consistency across inference stages, reflects real engineering obstacles developers face when building production systems.

The experimental findings that current FD-SLMs exhibit degraded performance across multiple rounds and evaluation dimensions has important implications for the AI industry. Organizations developing conversational AI systems must now account for multi-round robustness as a core requirement rather than an afterthought. This creates new benchmarking standards that developers will need to meet, potentially accelerating research into more robust model architectures and training methodologies.

The open-source release of MTR-DuplexBench code and data establishes a common evaluation standard for the research community. This standardization typically accelerates progress by enabling direct comparison of competing approaches and focusing development efforts on addressing identified weaknesses. Future iterations will likely expand evaluation dimensions and complexity levels as full-duplex systems mature toward commercial deployment.

• →MTR-DuplexBench introduces the first comprehensive multi-round evaluation framework specifically designed for full-duplex speech language models.
• →Current FD-SLMs demonstrate performance degradation across multiple conversation rounds and evaluation dimensions, indicating significant development challenges ahead.
• →The benchmark evaluates conversational quality, dialogue coherence, instruction-following, and safety rather than focusing exclusively on conversational metrics.
• →Open-source availability enables research standardization and accelerates community efforts to improve full-duplex model robustness.
• →This work addresses critical gaps between single-turn research evaluation and real-world multi-round deployment requirements for conversational AI.