Clinical Term Extraction using Open-Source Small Language Models

Researchers evaluated 26 open-source small language models for extracting clinical terms related to amyotrophic lateral sclerosis (ALS) from unstructured patient notes, finding that hybrid approaches combining rule-based methods with machine learning outperform either approach alone. The study demonstrates that modest-sized language models can handle specialized medical information extraction tasks without task-specific training, though traditional regex-based systems remain competitive for this application.

This research addresses a critical challenge in healthcare informatics: converting unstructured clinical documentation into machine-readable formats that enable downstream analysis and decision support. The ALS case study is particularly relevant given the disease's rapid progression and the need for precise tracking of functional decline across multiple clinical dimensions. By evaluating 26 different open-source models systematically, the researchers provide empirical evidence about the practical performance boundaries of small language models in medical contexts.

The healthcare sector has long struggled with information extraction from clinical notes, where manual curation remains common despite being labor-intensive and error-prone. Open-source small language models offer potential cost and privacy advantages over proprietary solutions, making this evaluation timely as healthcare organizations increasingly adopt AI infrastructure. The finding that no single model universally outperformed others across all metrics reflects the nuanced tradeoffs between precision and recall that plague real-world medical applications.

The critical insight from this work is that hybrid architectures—combining rule-based systems with machine learning approaches—represent a more pragmatic path forward than wholesale replacement of existing extraction pipelines. Regex-based methods excelled at recall while sacrificing precision; conversely, some SLMs showed stronger precision at the cost of missing cases. This suggests that production systems should strategically deploy different approaches for different term categories rather than adopting a single universal model.

The research points toward an evolving industry practice where open-source models serve as components within larger clinical workflows rather than standalone solutions. Future work should focus on understanding which term categories benefit most from each approach and developing efficient ensemble methods that maintain performance while reducing computational overhead.

• →Hybrid extraction systems combining regex rules with language models outperform either approach alone for clinical term detection.
• →Qwen3-4B-Instruct-2507 achieved the best performance among tested open-source small language models, indicating viable options exist beyond proprietary solutions.
• →Different models excel at different metrics, suggesting category-specific deployment strategies optimize extraction pipeline performance.
• →Open-source models demonstrated capability for specialized medical tasks without task-specific training data or fine-tuning.
• →Rule-based baselines remain competitive for clinical information extraction, challenging the assumption that neural approaches universally supersede traditional methods.