When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation

Researchers fine-tuned the Llama 2 7B model using real patient-doctor interaction transcripts to improve medical query responses, but found significant discrepancies between automatic similarity metrics and GPT-4 evaluations. The study highlights the challenges in evaluating AI medical models and recommends human medical expert review for proper validation.

• →Fine-tuning Llama 2 7B on medical dialogue transcripts showed improvements across most metrics except GPT-4 evaluation.
• →Automatic text similarity metrics disagreed with GPT-4's assessment of the model's medical performance.
• →LLMs often perform poorly in medical contexts and may provide harmful misguidance to users.
• →The research emphasizes the need for human medical expert evaluation rather than relying solely on automated metrics.
• →There are significant challenges in properly evaluating AI models for healthcare applications.