Attention Smoothing Is All You Need For Unlearning

Researchers propose Attention Smoothing Unlearning (ASU), a new framework that helps Large Language Models forget sensitive or copyrighted content without losing overall performance. The method uses self-distillation and attention smoothing to erase specific knowledge while maintaining coherent responses, outperforming existing unlearning techniques.

• →ASU addresses the critical problem of LLMs memorizing sensitive, copyrighted, or hazardous content that poses privacy and legal risks.
• →The method uses attention smoothing with increased softmax temperature to suppress lexical and semantic associations tied to memorized knowledge.
• →ASU outperforms baseline unlearning methods across multiple benchmarks including TOFU, MUSE, and WMDP.
• →The framework maintains model coherence and utility while successfully erasing targeted factual information.
• →The approach offers a computationally feasible alternative to retraining models from scratch.