A Dataset of Robot-Patient and Doctor-Patient Medical Dialogues for Spoken Language Processing Tasks

Researchers introduce MeDial-Speech, a new 111+ hour speech dataset for training medical AI systems to conduct patient consultations across four health conditions. The study benchmarks state-of-the-art LLMs including Claude Sonnet 4, GPT-5 mini, and DeepSeek-V3, revealing that while Claude Sonnet 4 achieves 71-75% accuracy in medical dialogue tasks, all models exhibit significant overconfidence in their probabilistic predictions.

The development of MeDial-Speech represents a meaningful step toward operationalizing conversational AI in clinical settings, addressing a critical gap between general-purpose language models and domain-specific medical applications. Medical dialogue systems require both linguistic accuracy and contextual understanding of patient symptoms, making specialized datasets essential for proper model evaluation and deployment.

The broader context reflects growing recognition that LLMs alone cannot reliably handle medical consultations without task-specific training and evaluation frameworks. Healthcare systems increasingly explore AI-assisted diagnosis and patient intake, but the liability and safety implications demand rigorous benchmarking. MeDial-Speech's inclusion of robot-patient and doctor-patient interactions provides realistic training scenarios that generic datasets cannot replicate, addressing a documented limitation in AI medical research.

For the AI industry, this work highlights both opportunities and limitations in enterprise AI deployment. Claude Sonnet 4's superior performance (74.7% with automatic transcription) suggests viable pathways for AI-assisted medical consultations, yet the finding that all tested LLMs exhibit overconfidence despite accuracy rates below 75% raises critical safety concerns. This overconfidence phenomenon could lead to false diagnostic confidence in real-world applications, a serious problem for healthcare systems. The dataset's free availability accelerates development but also underscores the need for standardized safety protocols before deployment.

Investors and healthcare organizations should monitor whether subsequent research addresses the overconfidence issue through calibration techniques or ensemble approaches. The next critical milestone involves real-world validation with actual patients and clinical outcomes, which will determine whether current LLM capabilities can genuinely support medical practice.

• →MeDial-Speech provides 111+ hours of medical dialogue speech data covering dementia, heart failure, shoulder pain, and angina for training specialized medical AI systems.
• →Claude Sonnet 4 achieves the highest accuracy (74.7%) in medical sentence selection benchmarks, outperforming GPT-5 mini and DeepSeek-V3.
• →All tested LLMs demonstrate problematic overconfidence in medical predictions regardless of answer correctness, creating potential safety risks for clinical deployment.
• →The dataset is freely available for non-commercial research, accelerating development of medical AI applications across the research community.
• →Real-world clinical validation remains the critical next step before medical dialogue systems can reliably support patient consultations in practice.