Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

Researchers introduce TeLAPA, a continual reinforcement learning framework that maintains diverse policy archives instead of relying on single-model preservation, addressing the loss of plasticity problem where retained policies fail to serve as effective starting points for rapid adaptation across new tasks.

The paper addresses a fundamental challenge in continual reinforcement learning: how agents can learn sequentially across multiple tasks without catastrophic forgetting or losing adaptability. Traditional approaches commit to preserving a single evolving policy, assuming this strategy maximizes knowledge reuse. However, the authors demonstrate this assumption fails—a previously successful policy often cannot rapidly readapt after experiencing task interference, a phenomenon called loss of plasticity that single-policy methods cannot resolve.

TeLAPA reimagines continual RL by applying insights from quality-diversity optimization, which seeks to preserve multiple high-performing solutions with behavioral diversity rather than converging to one optimal solution. The framework organizes policies into per-task archives while maintaining a shared latent space, enabling comparison and reusability despite non-stationary environment drift. This transforms the problem from preserving isolated champions to curating competent policy neighborhoods.

The approach has significant implications for practical AI systems that must adapt across dynamic environments—autonomous robotics, game-playing agents, and multi-task learning scenarios. Experiments on MiniGrid environments show TeLAPA learns more tasks successfully, recovers faster on revisited tasks, and maintains higher cumulative performance. Critically, the research reveals source-optimal policies often differ from transfer-optimal policies, meaning the best performer for one task rarely transfers best to new tasks.

Looking forward, this reframes how researchers should approach lifelong learning architectures. Rather than optimizing for single-policy preservation, systems should explicitly maintain diverse, behaviorally related alternatives. Future work likely extends this to larger-scale environments and explores how latent space alignment scales with task complexity and heterogeneity.

• →Single-policy preservation in continual RL fails to maintain plasticity, limiting rapid adaptation to new tasks
• →TeLAPA maintains diverse policy archives in a shared latent space, enabling faster recovery and better task performance
• →Source-optimal policies for one task rarely transfer optimally to new tasks, requiring retention of alternative candidates
• →Policy neighborhood diversity outperforms collapsing solutions into a single representative in non-stationary environments
• →This research shifts continual RL design from isolation optimization toward multi-candidate skill-aligned preservation