ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

Researchers introduce ICU-Bench, a new benchmark for testing machine unlearning in multimodal AI models, addressing privacy concerns from large-scale training datasets. The benchmark reveals that current unlearning methods struggle with continuous privacy deletion requests, highlighting a critical gap between theoretical approaches and real-world deployment needs.

ICU-Bench addresses a fundamental challenge emerging as multimodal large language models become ubiquitous: the need to selectively remove sensitive training data after deployment. This research benchmarks continual unlearning—the ability to process sequential privacy deletion requests—which differs from static unlearning scenarios typically studied in academic literature. The benchmark uses privacy-critical document data including medical reports and labor contracts, reflecting genuine regulatory pressures from GDPR, CCPA, and similar frameworks requiring data deletion capabilities.

The research demonstrates that existing unlearning methods exhibit significant limitations when handling sequential deletion tasks. Current approaches struggle to balance three competing objectives: effectively removing target information, preserving historical forgetting decisions, and maintaining model utility on retained data. This tension becomes acute in real deployments where companies face continuous deletion requests from users or regulatory bodies.

For the AI development community, ICU-Bench establishes objective evaluation criteria for a previously under-researched problem. The benchmark's structured metrics enable researchers to measure forgetting effectiveness and stability systematically, accelerating progress toward production-ready solutions. As multimodal models handle increasingly sensitive applications in healthcare, finance, and legal domains, robust unlearning mechanisms become competitive differentiators.

Looking ahead, organizations deploying MLLMs in regulated industries will pressure vendors to implement effective continual unlearning. The research signals that current solutions are insufficient, creating market incentives for specialized unlearning techniques. Success requires developing methods that efficiently handle sequential operations without catastrophic forgetting or utility degradation—a non-trivial engineering challenge that will likely influence future model architecture decisions.

• →ICU-Bench introduces the first comprehensive benchmark for evaluating continual unlearning in multimodal AI models using privacy-critical document data.
• →Existing machine unlearning methods fail to adequately balance forgetting quality, utility preservation, and scalability in continuous deletion scenarios.
• →The benchmark includes 1,000 privacy-sensitive profiles, 9,500 images, and 100 sequential forget tasks to simulate realistic deployment conditions.
• →Continual unlearning is essential for regulatory compliance in healthcare, legal, and financial domains handling sensitive multimodal data.
• →Current limitations in sequential unlearning create opportunities for specialized methods designed specifically for production AI systems.