When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

A comprehensive benchmark study reveals that properly calibrated rule-based autoscalers outperform six mainstream deep reinforcement learning algorithms on cost in adaptive resource control tasks. The research challenges assumptions about DRL superiority, identifying baseline calibration and reward engineering as greater bottlenecks than algorithm selection.

This research fundamentally questions the conventional wisdom that deep reinforcement learning automatically outperforms traditional control methods in resource allocation problems. The RLScale-Bench study, instantiated on Kubernetes infrastructure, demonstrates that engineering effort matters more than algorithmic sophistication—a finding with profound implications for practitioners deploying ML systems in production environments. The benchmark evaluates PPO, DQN, A2C, SAC, TD3, and DDPG across diverse workload patterns with rigorous evaluation protocols, revealing that discrete-action algorithms significantly outperform continuous variants due to action-space constraints, a technical detail often overlooked in academic comparisons. The research identifies a critical gap between theoretical ML research and practical deployment requirements. Organizations considering DRL investments for resource control should recognize that success depends primarily on meticulous baseline calibration and reward function design rather than selecting advanced algorithms. The finding that no single algorithm dominates across workload types suggests that specialized, simpler controllers may often provide better reliability and cost-efficiency than general-purpose learning approaches. This challenges the assumption that more sophisticated ML techniques automatically yield superior results. For cloud infrastructure operators and DevOps teams, the implications are clear: before investing in complex RL systems, establishing well-tuned rule-based baselines provides a stronger foundation and clearer performance benchmark. The distribution-shift generalization probes further highlight real-world complexity that academic benchmarks often ignore, suggesting DRL adoption requires stronger engineering rigor than currently practiced.

• →Calibrated rule-based controllers achieve lowest cost across all tested workloads, outperforming six mainstream DRL algorithms
• →Discrete-action algorithms reduce constraint violations by one to two orders of magnitude compared to continuous-action approaches
• →Algorithm selection proves less critical than baseline calibration, reward engineering, and evaluation protocol design
• →No single DRL algorithm dominates across different workload patterns, with performance rankings shifting significantly
• →The benchmark reveals that engineering fundamentals matter more than algorithmic sophistication in practical resource control