Dynamic one-time delivery of critical data by small and sparse UAV swarms: a model problem for MARL scaling studies

Researchers introduce a family of deterministic games designed to test Multi-Agent Reinforcement Learning (MARL) scalability for decentralized UAV swarm control tasked with relaying critical data. While baseline policies using Dijkstra's algorithm perform comparably to standard MARL algorithms for small agent counts, existing MARL approaches demonstrate significant scalability limitations as swarm size increases.

This research addresses a fundamental challenge in autonomous systems: coordinating multiple agents to accomplish distributed tasks without centralized control. The study applies Multi-Agent Reinforcement Learning to a practical problem—UAV swarms delivering critical data packages—and reveals critical gaps between current MARL capabilities and real-world deployment requirements. The researchers establish a controlled benchmark using deterministic games, enabling rigorous comparison between learning-based approaches and traditional algorithmic baselines.

The work fits within broader efforts to scale reinforcement learning across multiple heterogeneous agents operating in dynamic environments. Current MARL algorithms, while showing promise for small groups, struggle with computational complexity and coordination overhead as agent populations grow. This scalability bottleneck has been a persistent challenge limiting autonomous swarm applications in logistics, emergency response, and scientific missions.

For the robotics and autonomous systems industries, these findings indicate that production-grade swarm applications require either algorithmic breakthroughs or hybrid approaches combining learning with classical optimization. The competitive baseline using Dijkstra's shortest path suggests that traditional methods remain viable for well-defined problem spaces, potentially delaying MARL adoption in certain domains. Organizations developing swarm technologies must account for scaling limitations when planning multi-agent deployments beyond small pilot programs.

Future work will likely focus on algorithmic innovations to address the identified scaling issues, possibly through hierarchical coordination approaches or modified reward structures that reduce computational requirements. The publicly available code and benchmark enable community-driven improvements and standardized evaluation across different MARL frameworks.

• →Current MARL algorithms show competitive performance with baseline methods for small UAV swarms but fail to scale effectively with increased agent counts.
• →A deterministic game family is introduced as a standardized benchmark for evaluating MARL scalability in multi-agent coordination problems.
• →Classical algorithms using Dijkstra's shortest path remain competitive with reinforcement learning approaches for structured data-relay tasks.
• →Computational complexity and coordination overhead emerge as critical bottlenecks preventing MARL deployment in larger autonomous swarms.
• →Open-source implementation and visualizations support reproducible research and community development of improved MARL scaling solutions.