PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models

Researchers introduce PPU-Bench, a benchmark for testing personalized partial unlearning in multimodal AI models, addressing the challenge of selectively removing sensitive memorized information while preserving model utility. The study reveals significant trade-offs between forgetting target knowledge and retaining non-target facts, proposing Boundary-Aware Optimization as a solution for fine-grained factual control.

PPU-Bench addresses a critical gap in AI safety research by establishing the first real-world benchmark for personalized partial unlearning in vision-language models. Unlike previous synthetic approaches that oversimplify deletion requests, this benchmark reflects genuine scenarios where users need fine-grained control over what multimodal models remember about specific subjects. The research involves 24,000 samples from 500 public figures tested across three complexity levels, providing a comprehensive evaluation framework.

The findings expose a fundamental challenge in current unlearning methods: complete subject deletion often removes visual identity rather than specific facts, creating inefficient solutions that damage model capabilities unnecessarily. The forget-retain trade-off emerges as the central problem, where removing target knowledge frequently compromises the model's ability to retain related but distinct information. This complexity reflects real-world privacy requirements where deletion must be surgical rather than wholesale.

The proposed Boundary-Aware Optimization method represents meaningful progress by explicitly modeling the boundaries between facts an AI should forget versus retain within a single subject. This approach enables more nuanced unlearning that respects privacy needs without degrading model performance across the board. For AI developers and companies deploying multimodal models in sensitive domains, these findings clarify that effective unlearning requires sophisticated techniques beyond simple fine-tuning approaches.

Looking forward, the benchmark establishes standardized evaluation criteria for assessing unlearning methods rigorously. This work will likely accelerate development of more targeted unlearning techniques and influence how companies design privacy controls into AI systems. The robustness analysis revealing vulnerabilities under adversarial attacks suggests that future methods must consider attack resistance alongside forgetting effectiveness.

• →PPU-Bench introduces 24K multimodal samples to benchmark realistic personalized unlearning across complete, selective, and fine-grained deletion settings.
• →Complete unlearning often suppresses visual identity rather than factual knowledge, revealing inefficiencies in existing deletion approaches.
• →Significant forget-retain trade-offs exist in selective and personalized unlearning, exposing challenges in controlling intra-subject factual boundaries.
• →Boundary-Aware Optimization explicitly models forget-retain boundaries to enable more surgical, effective unlearning without degrading model utility.
• →Robustness analysis reveals distinct vulnerabilities across unlearning settings when exposed to cross-image and prompt-based attacks.