Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses

Researchers demonstrate that standard LLM-as-a-judge methods achieve only 52% accuracy in detecting hallucinations and omissions in mental health chatbots, failing in high-risk healthcare contexts. A hybrid framework combining human domain expertise with machine learning features achieves significantly higher performance (0.717-0.849 F1 scores), suggesting that transparent, interpretable approaches outperform black-box LLM evaluation in safety-critical applications.

The deployment of LLM-powered chatbots in mental health services presents a critical safety challenge that current evaluation methods inadequately address. Leading large language models tasked with judging other LLMs' responses achieve only 52% accuracy on mental health counseling data, with some hallucination detection approaches showing near-zero recall. This performance gap reveals a fundamental limitation: LLMs struggle to recognize the nuanced linguistic patterns and therapeutic principles that domain experts naturally identify, particularly when subtle errors could cause real harm to vulnerable users.

This research emerges from broader concerns about AI safety in high-stakes domains. While LLMs have become popular as automated judges due to their scalability and sophistication, their black-box nature and inability to ground knowledge in domain-specific expertise create dangerous blind spots. The gap between general-purpose AI capabilities and specialized healthcare requirements has become increasingly apparent as mental health applications proliferate.

The proposed hybrid framework addresses this by extracting five interpretable dimensions—logical consistency, entity verification, factual accuracy, linguistic uncertainty, and professional appropriateness—through human-LLM collaboration. Traditional machine learning models trained on these features substantially outperform pure LLM approaches, achieving 0.849 F1 on public benchmarks. This methodology has immediate implications for healthcare AI deployment, establishing a template for safety-critical applications where interpretability and human oversight matter more than automation speed. Organizations building mental health tools must prioritize explainable evaluation systems over convenient black-box solutions.

The research signals a shifting understanding of AI deployment: raw model capability cannot substitute for domain expertise in regulated or high-risk sectors. Future mental health chatbot development will likely require similar human-in-the-loop verification architectures.

• →Standard LLM judges achieve only 52% accuracy on mental health chatbot responses, demonstrating critical limitations in high-risk domains
• →Hybrid human-expert and machine learning frameworks achieve 0.849 F1 score versus LLM-only approaches, showing substantial performance improvement
• →Five interpretable analytical dimensions (consistency, verification, accuracy, uncertainty, appropriateness) outperform black-box LLM evaluation methods
• →Mental health applications require explicit domain expertise and transparency rather than relying solely on AI automation
• →This hybrid approach establishes a replicable safety framework for deploying LLMs in regulated healthcare and safety-critical sectors