Conformal Prediction for Risk-Controlled Medical Entity Extraction Across Clinical Domains

Researchers developed a conformal prediction framework for Large Language Models used in medical entity extraction, testing on FDA drug labels and radiology reports. The study found that model calibration varies significantly across clinical domains, with models being underconfident on structured data but overconfident on free-text reports, achieving 90% target coverage with 9-13% rejection rates.

• →LLMs show opposite confidence calibration patterns across different medical domains - underconfident on structured FDA labels, overconfident on free-text radiology reports.
• →The conformal prediction framework achieved target coverage of 90% or higher in both clinical settings with manageable rejection rates of 9-13%.
• →GPT-4.1 achieved 97.7% accuracy extracting entities from FDA drug labels across 128,906 evaluated entities.
• →Entity extraction F1 scores ranged from 0.81 to 0.84 when tested on radiology reports against physician annotations.
• →Model calibration is not a global property but depends on document structure, extraction category, and model architecture, requiring domain-specific approaches.