Offline Multi-agent Continual Cooperation via Skill Partition and Reuse

Researchers introduce COMAD, a framework for multi-agent reinforcement learning systems to continually discover and reuse coordination skills from offline data without catastrophic forgetting. The approach uses skill partitioning and density-based reusability estimation to enable agents to efficiently transfer knowledge across sequential tasks in open environments.

COMAD addresses a fundamental challenge in multi-agent reinforcement learning: how systems can learn transferable coordination skills from historical data while adapting to new tasks without performance degradation. Traditional approaches rely on fixed skill libraries designed through heuristics, which fail when tasks appear sequentially and skill spaces expand exponentially. This research tackles the dual problems of catastrophic forgetting—where learning new skills erases previous knowledge—and plasticity loss, where systems become too rigid to adapt.

The framework operates through two key mechanisms. First, it extracts coordination knowledge from mixed multi-agent behavior data using auto-encoders, converting raw behavioral patterns into reusable skills. Second, it employs multi-head policy architectures with a density-based reusability estimator that explicitly guides advantage functions toward the most relevant skills for each task. The theoretical backing demonstrates that COMAD approximates optimal solutions for continual skill discovery problems.

For the broader AI and machine learning field, this work carries significant implications for autonomous systems operating in dynamic environments. Multi-agent coordination represents a critical capability for swarm robotics, autonomous vehicle fleets, and distributed computing systems. The ability to continually expand skill libraries while maintaining performance on previous tasks directly enhances system scalability and real-world applicability.

The empirical validation across multiple MARL benchmarks shows superior forward and backward transfer metrics—meaning new tasks benefit from prior learning while previous performance remains stable. This positions COMAD as a meaningful step toward more adaptable, generalizable multi-agent systems suitable for practical deployment in evolving operational contexts.

• →COMAD enables continual skill discovery in multi-agent systems without catastrophic forgetting through skill partitioning and density-based reusability estimation.
• →The framework uses auto-encoders to extract coordination knowledge from offline multi-agent data and transform it into reusable skills.
• →Multi-head policy architectures with skill-augmented objectives guide agents toward the most relevant coordination patterns for sequential tasks.
• →Theoretical analysis confirms COMAD approximates optimal solutions for continual multi-agent skill discovery problems.
• →Empirical results demonstrate superior forward and backward transfer across diverse MARL benchmarks compared to existing baseline methods.