MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs

Researchers introduce MLUBench, a large-scale benchmark for evaluating lifelong unlearning in multimodal large language models (MLLMs), revealing that existing methods suffer from cumulative degradation. The study identifies a unique challenge in MLLM unlearning: removing data from one modality can damage the model's multimodal alignment, and proposes LUMoE as a solution to mitigate this degradation.

The emergence of MLUBench addresses a critical gap in AI safety and data governance as multimodal models become increasingly prevalent. While previous research focused on unlearning in unimodal systems, this work tackles the substantially more complex problem of sequential data removal requests in models that integrate vision and language. This matters because real-world scenarios involve continuous, unpredictable removal requests from data subjects exercising privacy rights, yet most existing benchmarks use static datasets, failing to reflect this dynamic reality.

The research reveals a fundamental architectural vulnerability in current MLLMs: the tight coupling between visual and textual representations means that aggressively unlearning from one modality cascades into performance degradation across the entire system. This finding has significant implications for compliance with emerging data protection regulations like GDPR, where organizations must balance user privacy rights against maintaining model performance. The proposed LUMoE method represents progress toward practical unlearning solutions, though the persistence of cumulative degradation across baselines suggests the problem remains unsolved.

For developers building multimodal AI systems, this research signals the need to rethink model architecture to support modular, non-destructive unlearning. Organizations training or deploying MLLMs should anticipate that current methods cannot reliably handle sequential privacy requests without quality loss. The open-sourced MLUBench dataset enables broader community participation in solving this challenge, potentially accelerating development of more robust unlearning techniques that preserve model utility.

• →MLUBench introduces the first large-scale benchmark with 127 entities across 9 classes for evaluating sequential unlearning in multimodal models.
• →Existing unlearning methods exhibit severe cumulative degradation when applied repeatedly, making them impractical for real-world deployment.
• →Multimodal alignment constraints create unique challenges in MLLMs that don't exist in unimodal models, as unlearning from one modality damages overall performance.
• →LUMoE method significantly mitigates degradation but doesn't fully solve the problem, indicating further research is needed.
• →Open-sourced code and dataset enable community-wide research on a critical gap in AI safety and privacy compliance.