Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies

Researchers demonstrate that Evolution Strategies (ES) can effectively fine-tune large language models without catastrophic forgetting of prior tasks, contrary to recent concerns. By introducing Anchored Weight Decay (AWD), a regularization technique that constrains optimization toward initial parameters, the work shows ES-based continual learning is viable and computationally efficient compared to reinforcement learning approaches.

The research addresses a critical bottleneck in deploying large language models across multiple tasks sequentially. ES has emerged as an attractive alternative to reinforcement learning for LLM fine-tuning due to its simplicity and inference-only training approach, but recent findings suggested it caused irreversible performance degradation on previously learned tasks. This work fundamentally reframes the problem: prior-task performance drift during ES training is not permanent forgetting but rather a reversible phenomenon tied to the optimization dynamics of ES itself, particularly its tendency toward random walks in unconstrained weight-space directions.

The introduction of Anchored Weight Decay provides an elegant solution by anchoring parameter updates toward the initial model weights, effectively serving as a regularization boundary. This technique delivers comparable benefits to using substantially larger ES population sizes while dramatically reducing computational overhead—a significant efficiency gain for organizations scaling LLM deployments.

For the AI development community, this research validates ES as a legitimate continual learning approach, potentially enabling more cost-effective multi-task model adaptation. The findings have immediate implications for practitioners developing AI systems that must maintain performance across diverse tasks without expensive retraining cycles. The work also contributes fundamental insights about optimization dynamics in high-dimensional weight spaces, advancing theoretical understanding of how modern neural networks behave during training.

Future research will likely focus on combining AWD with other continual learning techniques and testing its effectiveness on increasingly large model scales and complex task sequences.

• →Prior-task forgetting in ES fine-tuning is reversible performance drift, not irreversible forgetting, and recovers during training.
• →Anchored Weight Decay (AWD) regularization constrains parameter updates toward initial weights, stabilizing performance on prior tasks.
• →AWD achieves computational efficiency gains equivalent to much larger ES population sizes at significantly lower cost.
• →Performance drift occurs in both ES and RL fine-tuning, indicating it is a general challenge rather than ES-specific.
• →ES is positioned as a viable approach for continual learning in LLMs when equipped with appropriate regularization techniques.