Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning

Researchers propose QGF (Q-Guided Flow), a reinforcement learning algorithm that optimizes policies entirely at test time using value gradients to guide pre-trained flow models, avoiding the training instability issues of traditional actor-critic approaches while maintaining competitive performance on offline RL benchmarks.

QGF represents a meaningful shift in how researchers approach reinforcement learning with expressive policy models like diffusion and flow policies. Rather than integrating these models into RL training pipelines—a process that typically introduces instability and requires specialized objectives—the proposed method performs all policy optimization at inference time, leaving supervised pretraining untouched. This architectural choice addresses a genuine bottleneck in scaling imitation learning for robotics and control tasks.

The motivation stems from years of research showing that backpropagating through denoising processes destabilizes learning while increasing computational overhead. By decoupling policy improvement from policy learning, QGF sidesteps these known failure modes. The algorithm pretrains a flow policy via standard behavioral cloning and a value function, then at test time uses value gradients to steer the reference policy toward higher-value actions. This test-time optimization approach has precedent in diffusion model research but remains underexplored in RL contexts.

The empirical results demonstrate meaningful advantages: QGF outperforms existing test-time RL methods on single-task and goal-conditioned offline RL benchmarks while remaining competitive with state-of-the-art training-time algorithms at substantially lower computational cost. Critically, the method exhibits favorable scaling properties by avoiding actor-critic training instability, suggesting it could handle larger model architectures more robustly than current approaches.

For the robotics and embodied AI communities, this work offers a practical alternative that doesn't sacrifice performance for stability. The approach is particularly valuable for real-world applications where training instability translates directly into unsafe behaviors. Future research should explore whether this paradigm extends to online RL settings and whether value gradient guidance generalizes across diverse task distributions.

• →QGF performs all policy optimization at test time using value gradients, avoiding the training instability of integrating flow models into RL pipelines
• →The method matches or exceeds state-of-the-art training-time RL algorithms while requiring significantly less computational resources
• →Test-time optimization of pre-trained policies addresses a key bottleneck in scaling imitation learning for high-dimensional control tasks
• →The approach exhibits better scaling properties with model size by eliminating actor-critic training instability
• →QGF demonstrates strong performance on offline RL benchmarks with high-dimensional action spaces, suggesting practical applicability to real robotics