When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

Researchers introduce Q2RL, a novel algorithm that combines behavior cloning with reinforcement learning to enable robots to improve their policies through online interaction. The method uses Q-value estimation and gating mechanisms to prevent policy degradation from distribution mismatch, achieving 100% success rates on complex manipulation tasks in 1-2 hours of real robot learning.

Q2RL addresses a fundamental challenge in robot learning: the gap between learning from demonstrations and continuous online improvement. Behavior cloning excels at replicating expert actions but lacks mechanisms for self-directed improvement, while naive reinforcement learning often abandons valuable learned behaviors when encountering new data distributions. This research bridges that gap through an elegant two-stage approach that leverages the strengths of both paradigms.

The broader context reflects years of development in offline-to-online learning, where researchers have struggled with policy collapse caused by distribution shift between offline datasets and online environments. Previous methods have shown mixed results, often reverting to suboptimal behaviors. Q2RL's innovation lies in its Q-Gating mechanism, which acts as a decision arbiter—selecting actions from either the BC policy or RL policy based on their estimated Q-values. This preserves good behaviors while enabling exploration toward improvements.

For robotics development and autonomous systems, this represents meaningful progress in reducing training time for physical tasks. Achieving robust policies for high-precision manipulation in 1-2 hours of actual robot interaction—rather than days or weeks—substantially lowers deployment costs and accelerates iteration cycles. The demonstrated improvements across D4RL benchmarks and real robotic tasks like pipe assembly suggest practical viability beyond academic settings.

Looking forward, the technique's efficiency makes it particularly relevant for industrial robotics and in-field adaptation scenarios. Researchers should monitor whether similar Q-gating principles apply to other domains like locomotion or multi-agent systems, and whether the approach scales to longer deployment horizons where additional online learning becomes necessary.

• →Q2RL enables robots to improve behavior-cloned policies through online reinforcement learning while preventing distribution-mismatch degradation
• →Q-Gating mechanism intelligently selects between BC and RL policy actions based on Q-value estimates, preserving good behaviors while enabling improvement
• →Achieved 100% success rates on complex contact-rich manipulation tasks with only 1-2 hours of real robot interaction
• →Outperforms state-of-the-art offline-to-online baselines on both success rate and convergence speed across multiple benchmarks
• →Practical algorithm design demonstrates viability for industrial robotics applications requiring rapid, efficient policy adaptation