Orthogonal Subspace Projection for Continual Machine Unlearning via SVD-Based LoRA

Researchers propose an SVD-based orthogonal subspace projection method for continual machine unlearning that prevents interference between sequential deletion tasks in neural networks. The approach maintains model performance on retained data while effectively removing influence of unlearned data, addressing a critical limitation of naive LoRA fusion methods.

This research tackles a fundamental challenge in machine learning: enabling models to selectively forget data without degrading overall performance. As privacy regulations and user rights around data deletion become increasingly stringent, continual unlearning represents a technically and commercially important problem. The paper demonstrates that naive approaches to sequential unlearning suffer from parameter collision—when multiple low-rank updates accumulate, they interfere with each other, causing catastrophic forgetting of previously retained knowledge.

The proposed SVD-guided orthogonal projection method represents an elegant solution by constraining new unlearning updates to operate in subspaces orthogonal to previous updates. This geometric approach ensures task isolation without requiring dynamic routing mechanisms at inference time, making deployment simpler and more efficient. The experimental results are striking: while baseline fusion methods degrade retained accuracy from 60.39% to 12.70% after 30 sequential unlearning operations, the proposed method maintains approximately 58.1% accuracy while preserving unlearning efficacy.

For the AI industry, this work has significant implications for model governance and regulatory compliance. Organizations managing large-scale models will benefit from efficient, principled methods to honor user deletion requests without retraining models from scratch. The static fusion approach also reduces computational overhead compared to dynamic routing alternatives, making large-scale unlearning more economically viable.

Looking forward, researchers should investigate how this method scales to transformer architectures and larger models commonly used in production systems. Integration of these techniques into standard fine-tuning frameworks could accelerate adoption across the industry.

• →SVD-based orthogonal projection prevents parameter collision between sequential unlearning tasks without dynamic routing overhead
• →Method maintains 58.1% retained accuracy across 30 unlearning operations compared to 12.70% for baseline fusion approaches
• →Static fusion design simplifies deployment while preserving both unlearning efficacy and model utility on retained data
• →Addresses critical need for efficient, scalable machine unlearning as privacy regulations increasingly require data deletion capabilities
• →Approach applicable to LoRA-based fine-tuning, a widely-adopted method in modern deep learning