Activation Function Design Sustains Plasticity in Continual Learning

Researchers from arXiv demonstrate that activation function design is crucial for maintaining neural network plasticity in continual learning scenarios. They introduce two new activation functions (Smooth-Leaky and Randomized Smooth-Leaky) that help prevent models from losing their ability to adapt to new tasks over time.

• →Activation function choice significantly impacts a model's ability to maintain plasticity in continual learning scenarios.
• →Traditional i.i.d. training benchmarks don't reveal the importance of activation functions that becomes apparent in continual learning.
• →Two new activation functions (Smooth-Leaky and Randomized Smooth-Leaky) were developed to mitigate plasticity loss.
• →The research provides a lightweight, domain-general approach to sustaining model adaptability without requiring extra capacity.
• →The findings apply across both supervised learning and reinforcement learning environments with distribution shifts.