Continual Learning in Large Language Models: Methods, Challenges, and Opportunities

This comprehensive survey examines continual learning methodologies for large language models, focusing on three core training stages and methods to mitigate catastrophic forgetting. The research reveals that while current approaches show promise in specific domains, fundamental challenges remain in achieving seamless knowledge integration across diverse tasks and temporal scales.

• →Continual learning enables LLMs to adapt to evolving knowledge while preventing catastrophic forgetting from static pre-training paradigms.
• →The survey structures CL methodologies around three core stages: continual pre-training, fine-tuning, and alignment.
• →Research covers rehearsal-based, regularization-based, and architecture-based methods with distinct forgetting mitigation mechanisms.
• →Current methods demonstrate promising results in specific domains but face challenges in seamless knowledge integration.
• →The work provides a structured framework for understanding achievements and future opportunities in lifelong learning for language models.