Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning

Researchers introduce SETA, a machine learning framework that addresses catastrophic forgetting in large language models through sparse expert decomposition. The method separates task-specific and shared knowledge into distinct expert modules, enabling models to retain previous capabilities while learning new ones—a fundamental challenge in continual AI development.

The plasticity-stability dilemma represents one of artificial intelligence's most persistent technical obstacles. Models face inherent tension between adapting to new information and preserving learned knowledge; updating parameters for novel tasks typically erases previously acquired capabilities. SETA tackles this by implementing adaptive sparse subspace decomposition, fundamentally restructuring how language models store and retrieve knowledge across sequential learning phases.

Continual learning has gained prominence as enterprises deploy AI systems that must evolve without retraining from scratch. Previous approaches treated all model parameters uniformly, forcing tasks to compete for the same computational resources. SETA's innovation lies in its dual-expert architecture: task-specific experts isolate domain-particular patterns while shared experts capture generalizable features. The routing-aware regularization mechanism protects knowledge at both weight and routing levels, automatically selecting appropriate expert combinations during inference.

Experimental validation across domain-specific benchmarks demonstrates measurable improvements in early-task retention and backward transfer on LLaMA-2 7B and Qwen3-4B models. This indicates the framework addresses real operational constraints faced by organizations deploying sequential learning systems. Superior backward transfer—where learning new tasks enhances performance on earlier ones—suggests the shared expert mechanism genuinely identifies transferable patterns rather than merely segregating knowledge.

The framework's effectiveness on production-scale models carries implications for AI infrastructure efficiency. Organizations training or fine-tuning models could reduce computational overhead and training iterations required for multi-domain deployment. However, practical adoption depends on implementation complexity and whether performance gains justify additional architectural overhead compared to simpler continual learning baselines.

• →SETA uses sparse expert decomposition to separate task-specific and shared knowledge, directly addressing catastrophic forgetting in continual learning.
• →Routing-aware regularization protects knowledge at both weight and routing levels, enabling stable parameter updates across sequential learning tasks.
• →Experimental results show competitive performance with improved retention of early-task knowledge and backward transfer on LLaMA-2 7B and Qwen3-4B models.
• →The framework automates expert selection during inference through a unified gating network, reducing manual model configuration requirements.
• →Practical applications include more efficient fine-tuning pipelines and reduced computational overhead for multi-domain AI deployment.