MedThink: Enhancing Diagnostic Accuracy in Small Models via Teacher-Guided Reasoning Correction

MedThink presents a two-stage knowledge distillation framework that improves diagnostic accuracy in smaller language models by having teacher LLMs guide reasoning correction rather than simply transferring surface-level patterns. The approach achieves up to 12.7% improvement over baseline models while maintaining computational efficiency for resource-constrained clinical environments.

MedThink addresses a fundamental challenge in deploying clinical AI: large language models excel at medical reasoning but remain computationally prohibitive for hospitals and clinics with limited infrastructure. Traditional knowledge distillation compresses LLMs into smaller models but sacrifices the structured reasoning chains essential for reliable diagnosis. This work introduces a more sophisticated distillation methodology that treats reasoning as the core asset to transfer rather than mere answer patterns.

The framework's two-stage approach reflects a deeper understanding of how domain expertise operates. The first stage establishes foundational knowledge through teacher-guided fine-tuning with domain explanations. The second stage targets the model's reasoning failures directly—when the student model produces incorrect diagnoses, the teacher generates explicit reasoning chains showing how correct medical knowledge connects to proper answers. This iterative correction process mirrors how human medical educators train clinicians.

The experimental results across general medical benchmarks and a specialized gastroenterology dataset demonstrate that reasoning-focused distillation substantially outperforms six competing distillation strategies. The 56.4% accuracy on gastroenterology tasks and 12.7% improvement margins suggest this methodology has practical clinical value. The work enables smaller models to operate in settings where deploying GPT-4 or similar systems remains infeasible due to cost, latency, or regulatory constraints.

Looking ahead, this approach could accelerate healthcare AI adoption in resource-limited regions and smaller institutions. The publicly available code creates opportunities for broader implementation and refinement. Future work may explore whether reasoning-centric distillation applies to other technical domains requiring specialized knowledge and reliable inference chains.

• →MedThink's two-stage distillation framework prioritizes reasoning chain transfer over superficial pattern matching in clinical AI models.
• →Small language models equipped with teacher-guided reasoning correction outperform traditional distillation approaches across multiple benchmarks.
• →The methodology achieves 56.4% accuracy on specialized gastroenterology tasks while maintaining computational efficiency for deployment in resource-constrained settings.
• →Reasoning-focused distillation enables reliable clinical decision-support systems on edge devices without requiring massive computational infrastructure.
• →Open-source release of code and data accelerates adoption and enables further refinement of knowledge distillation techniques for specialized domains.