On the importance of multiple training seeds for evaluating machine unlearning

A new study reveals that evaluating machine unlearning algorithms requires multiple training seeds, not just multiple unlearning seeds from a single trained model, as unlearning performance varies significantly based on initial training conditions. This finding challenges current evaluation practices in machine unlearning research across image classification, federated learning, and large language models.

Machine unlearning—the process of removing a model's learned influence from specific data points without full retraining—has emerged as a critical capability for privacy-preserving machine learning. The research identifies a fundamental methodological flaw in how the field currently validates unlearning algorithms. By analyzing experiments across multiple domains, the authors demonstrate that relying on a single training seed produces unreliable performance estimates, particularly problematic for deterministic unlearning methods that yield identical results from the same starting model.

The broader context involves growing regulatory pressure around data privacy and the right to be forgotten, making efficient unlearning algorithms increasingly important for AI systems handling sensitive information. Current industry practice assumes varying unlearning seeds adequately tests algorithm robustness, but this study proves that training seed variation significantly impacts results. The findings extend beyond academic image classification into practical applications including federated learning systems and large language models, suggesting the problem pervades modern AI development.

For AI researchers and practitioners, this work necessitates rethinking experimental protocols and resource allocation. Conducting experiments with multiple training seeds requires substantially more computational resources and time, imposing practical constraints on research throughput. The authors provide guidance on optimal seed selection, helping teams balance statistical rigor against computational costs. This standardization could prevent misleading performance claims and ensure unlearning algorithms are properly validated before deployment in production systems handling user data.

• →Machine unlearning evaluation requires multiple training seeds to produce reliable performance assessments, not just multiple unlearning runs from a single model
• →Performance variation across training seeds is significant in image classification, federated learning, and large language model unlearning tasks
• →Deterministic unlearning methods are particularly sensitive to training seed selection since they produce identical outputs from identical starting conditions
• →Increasing unlearning seeds cannot compensate for using insufficient training seeds, making both dimensions essential for proper evaluation
• →The research provides specific guidance for selecting optimal numbers of training and unlearning seeds to balance statistical validity with computational costs