DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher

Researchers propose DUET, a new distillation-based method for LLM unlearning that removes undesirable knowledge from AI models without full retraining. The technique combines computational efficiency with security advantages, achieving better performance in both knowledge removal and utility preservation while being significantly more data-efficient than existing methods.

• →DUET addresses major limitations of existing LLM unlearning methods including computational overhead and vulnerability to attacks.
• →The method uses a student-teacher distillation approach where the student learns to refuse generating undesirable knowledge while preserving general capabilities.
• →Extensive benchmarks show DUET achieves superior performance in both forgetting unwanted knowledge and maintaining utility.
• →The technique is orders of magnitude more data-efficient than current state-of-the-art unlearning methods.
• →This advancement contributes to building more trustworthy AI systems by enabling selective knowledge removal.