Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Researchers introduce OMAD, an online multi-agent reinforcement learning framework that integrates diffusion-based generative models for improved policy coordination. The method achieves 2.5-5x improvements in sample efficiency across benchmark tasks by using relaxed policy objectives and joint distributional value functions to enable effective exploration without requiring tractable likelihood calculations.

This research addresses a significant gap in multi-agent reinforcement learning by applying diffusion models—a proven approach in generative AI—to the online learning setting. The core contribution lies in solving the entropy-based exploration problem that typically plagues diffusion models in RL contexts. Traditional diffusion models suffer from intractable likelihoods, which prevents standard entropy regularization techniques. OMAD circumvents this limitation through a scaled joint entropy maximization approach, enabling agents to explore effectively while maintaining coordination.

The framework builds on established MARL principles, specifically the centralized training with decentralized execution paradigm, but enhances it with distributional value functions that guide diffusion policy updates. This architecture allows multiple agents to learn coordinated behaviors efficiently—a persistent challenge in multi-agent systems where the action space explodes combinatorially. The 2.5-5x sample efficiency improvements represent substantial practical gains, as sample efficiency directly impacts training costs and real-world deployment feasibility.

For the AI research community, this work validates diffusion models' broader applicability beyond image generation and offline learning. The methodology could inspire similar applications in robotic control, autonomous systems, and cooperative task planning. The improvements in sample efficiency have concrete implications for practitioners seeking to reduce computational overhead in training multi-agent systems.

Future developments may explore scaling OMAD to larger agent populations, extending to heterogeneous agent teams, or applying the framework to real-world robotics problems where sample efficiency directly translates to reduced development costs and faster deployment timelines.

• →OMAD framework integrates diffusion models into online multi-agent RL for the first time with practical effectiveness
• →Relaxed policy objectives using scaled joint entropy enable exploration without tractable likelihood constraints
• →Joint distributional value functions ensure stable coordination across decentralized diffusion policies
• →Sample efficiency improvements of 2.5-5x reduce computational requirements for multi-agent training
• →Framework demonstrates state-of-the-art performance across 10 diverse benchmark tasks in MPE and MAMuJoCo