When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning

Researchers conduct a comprehensive benchmarking study of expert-guided reinforcement learning methods, revealing three critical failure modes that single-paper evaluations miss. They propose a decision rule based on pre-training observables to guide method selection, introducing EDGE as a new design point that exposes exploitable architectural dimensions.

This research addresses a significant gap in reinforcement learning evaluation methodology. While many RL papers propose using suboptimal expert controllers (PIDs, hand-designed gaits) to accelerate learning, each method has been studied in isolation on different benchmarks with inconsistent evaluation protocols. The authors standardize evaluation across multiple expert-guided RL methods using identical SAC backbones, hyperparameter optimization, and extensive seeding (100/50 seeds per environment), then systematically degrade expert quality through undertuning, action bias, and observation noise. This rigorous approach uncovers three failure modes invisible to single-paper studies: critic blind spots under certain bootstrapping mechanisms that paradoxically underperform baseline SAC, saturation effects on suboptimal experts, and buffer poisoning during expert handoff under deployment conditions. The research reveals no single method dominates—each excels in specific task regimes while failing predictably elsewhere. Notably, the study identifies a "RL-near-ceiling" regime where experts approach optimal performance, and none of the tested query-time methods successfully clear this barrier within computational budgets, raising fundamental questions about method scalability. The authors translate these findings into a testable decision rule indexed on three observable pre-training metrics: expert quality, task termination structure, and perturbation type. They introduce EDGE, a softmax-over-ensemble lower-confidence-bound design demonstrating that both gate form and scoring rule choices create exploitable trade-offs. This work contributes a reusable benchmark, comprehensive taxonomy, and practical decision framework rather than proposing a single superior method, making it valuable infrastructure for future expert-guided RL research.

• →Expert-guided RL methods fail in predictable ways that single-paper evaluations miss, including critic blind spots, residual saturation, and buffer poisoning.
• →No query-time expert method successfully improves performance when experts approach optimal solution quality within standard computational budgets.
• →A three-variable decision rule based on expert quality, task termination type, and perturbation structure can guide method selection across task regimes.
• →EDGE architecture demonstrates that both gating mechanisms and scoring rules represent individually exploitable design dimensions.
• →Standardized benchmarking with consistent hyperparameter optimization and extensive seeding (100/50 seeds) reveals dynamics invisible to isolated method papers.