Large Language Models for Outpatient Referral: Problem Definition, Benchmarking and Challenges

Researchers have developed a comprehensive evaluation framework for Large Language Models applied to outpatient referral systems in healthcare, revealing that LLMs offer limited advantages over simpler BERT-like models in static referral tasks but demonstrate potential in interactive dialogue scenarios. The study addresses the absence of standardized evaluation criteria for assessing LLM effectiveness in dynamic healthcare settings.

This research tackles a critical gap in healthcare AI deployment by establishing formal evaluation methodologies for LLM-based referral systems. The distinction between static and dynamic evaluation approaches reflects real-world clinical workflows where initial referral decisions often require iterative refinement through doctor-patient interactions. The finding that LLMs underperform relative to BERT-like models in predefined referral tasks suggests that model complexity does not automatically translate to superior performance in specialized medical domains with constrained decision spaces.

The healthcare sector has increasingly adopted LLMs for administrative and clinical tasks without rigorous comparative benchmarking against established baselines. This study provides empirical evidence that simpler, domain-adapted models may be more cost-effective and interpretable for specific healthcare applications. The framework's emphasis on interactive dialogue evaluation recognizes that modern healthcare systems require conversational AI that can ask clarifying questions and refine recommendations—a capability where LLMs appear to excel.

For healthcare AI developers and institutions implementing outpatient referral systems, this research suggests careful vendor evaluation and model selection based on task-specific benchmarking rather than general-purpose LLM capabilities. The standardized evaluation framework enables comparative assessment, reducing procurement risk and improving system transparency for clinical stakeholders. Organizations may benefit from hybrid approaches combining BERT-like models for initial referral triage with LLM-powered dialogue modules for refinement.

Future research should explore whether fine-tuned LLMs on specialized medical datasets can narrow the performance gap, and whether the interactive advantages justify the computational overhead and cost differential compared to baseline models.

• →LLMs show limited performance advantages over BERT-like models for static outpatient referral prediction tasks.
• →A new evaluation framework distinguishes between static referral accuracy and dynamic refinement through iterative dialogue.
• →LLMs demonstrate strength in asking clarifying questions during interactive clinical workflows.
• →Standardized evaluation criteria for healthcare AI systems are essential for informed model selection and deployment.
• →Simpler, domain-adapted models may offer better cost-effectiveness than general-purpose LLMs for constrained medical tasks.