RadOT-Eval: Auditable Structured-Evidence Transport for Radiology Report Evaluation

RadOT-Eval is a new AI framework that uses optimal transport algorithms to automatically evaluate radiology report generation by decomposing reports into structured clinical evidence units and detecting specific error types like omissions, hallucinations, and polarity reversals. The method achieves higher correlation with clinician-annotated errors than existing metrics and LLM-based evaluators, providing an auditable approach for quality assurance in high-stakes medical AI applications.

RadOT-Eval addresses a critical gap in AI evaluation for clinical applications where traditional similarity metrics fail to capture medically meaningful errors. Radiology report generation represents a particularly high-stakes use case because inaccuracies directly impact patient safety—a hallucinated finding or inverted polarity can fundamentally change clinical decisions. The framework's innovation lies in decomposing unstructured text into structured clinical evidence units before alignment, mirroring how radiologists actually reason about reports.

This research emerges from growing recognition that generic language model evaluators inadequately assess domain-specific text generation. Standard metrics like BLEU or ROUGE measure surface-level similarity, missing clinically critical differences such as location changes or uncertainty mismatches. RadOT-Eval's entropy-regularized optimal transport approach provides mathematical rigor for this matching problem, while its risk model quantifies clinically significant errors separately from minor discrepancies.

The practical implications extend beyond radiology. Any high-stakes text generation domain—legal contracts, financial reports, scientific abstracts—faces similar evaluation challenges where errors compound in severity. The demonstrated superiority over open-source LLM evaluators (GREEN-radllama2-7B) suggests structured approaches outperform black-box alternatives when clinical validity matters.

Future development should focus on broader clinical applications and integration with production radiology systems. The frozen evaluation protocol—trained only on ReXVal, tested on independent RadEvalX—establishes proper machine learning hygiene that institutional review boards demand. Wider adoption depends on validating performance across diverse radiology modalities and institutional settings.

• →RadOT-Eval achieves 0.715 Spearman correlation with clinician-annotated errors, outperforming standard metrics and existing LLM evaluators.
• →The framework decomposes reports into structured clinical evidence units to detect specific error types critical for medical safety.
• →Optimal transport alignment provides mathematically rigorous matching of clinical findings between reference and generated reports.
• →Frozen evaluation protocol on independent datasets demonstrates rigorous validation standards appropriate for clinical applications.
• →Structured evidence transport offers a generalizable approach for auditable evaluation in other high-stakes text generation domains.