REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference

Researchers introduce REMEDI, a benchmark for evaluating machine unlearning methods in clinical disease inference using real patient data from MIMIC-III. The study reveals fundamental trade-offs between model utility and data removal effectiveness, with existing unlearning techniques proving poorly suited for multi-label medical classification tasks.

REMEDI addresses a critical gap in machine learning research where privacy-preserving unlearning methods lack rigorous evaluation in high-stakes medical domains. The benchmark tackles the practical problem of removing patient data from trained models without retraining from scratch—a requirement driven by privacy regulations and patient consent withdrawals. Traditional unlearning benchmarks rely on synthetic datasets that fail to capture real-world complexities like label correlations and longitudinal patient records, limiting their applicability to healthcare systems handling sensitive information.

The work builds on growing concern within AI development communities regarding data privacy and regulatory compliance. As clinical language models become more prevalent in healthcare infrastructure, the ability to selectively forget patient information becomes essential for HIPAA compliance and ethical AI deployment. Current unlearning methods were designed primarily for image classification and text domains, leaving their effectiveness in medical AI largely unexplored.

The findings reveal a fundamental challenge: existing unlearning approaches struggle to balance forgetting specific patient data while maintaining model performance on retained data. Multi-label classification—where patients often have multiple concurrent diagnoses—particularly exposes these limitations. This tension between privacy and utility has significant implications for healthcare organizations deploying AI systems, as overly aggressive unlearning could degrade diagnostic accuracy, while insufficient unlearning leaves privacy vulnerabilities.

The public release of REMEDI enables standardized evaluation of new unlearning methods specifically designed for medical contexts. Future development in this space could drive innovation in privacy-preserving AI, potentially enabling broader adoption of clinical machine learning systems with stronger data governance frameworks.

• →REMEDI provides the first comprehensive benchmark for evaluating machine unlearning specifically in multi-label clinical disease inference using real MIMIC-III patient data.
• →Existing unlearning methods demonstrate fundamental trade-offs between preserving model utility and achieving complete data removal, unsuitable for production medical systems.
• →Multi-label and multi-class classification tasks in healthcare present unique unlearning challenges that synthetic-data benchmarks fail to capture.
• →The research highlights a critical gap between privacy requirements and technical capabilities in clinical AI deployment.
• →Standardized evaluation frameworks could accelerate development of medical-specific unlearning techniques meeting both regulatory and performance requirements.