Zero-Shot Off-Policy Learning

Researchers present a novel off-policy learning method that addresses distributional shift and value overestimation in zero-shot reinforcement learning by establishing a theoretical connection between successor measures and stationary density ratios. The approach enables agents to adapt to new tasks without additional training by inferring optimal importance sampling ratios on-the-fly, with successful benchmarks across motion tracking, continuous control, and long-horizon tasks.

This research tackles a fundamental challenge in reinforcement learning: enabling agents to learn optimal policies from fixed datasets and generalize to new tasks without retraining. Off-policy learning has long struggled with distributional shift—where the learned policy diverges from the data collection policy—and value function overestimation that compounds errors during decision-making. The zero-shot setting amplifies these difficulties by requiring immediate adaptation to novel tasks without any task-specific training data.

The key innovation lies in discovering that successor measures, which estimate state visitation frequencies, connect mathematically to stationary density ratios. This theoretical bridge allows the algorithm to compute optimal importance sampling weights that automatically correct for distributional mismatch and apply the best policy for any new task dynamically. Rather than learning task-specific policies, the method learns generalizable value estimates that transfer across different objectives.

The practical implications are substantial for robotics and autonomous systems. Agents could deploy pre-trained models that instantly adapt to new environments or goals—critical for real-world applications where retraining isn't feasible. The successful benchmarks on SMPL Humanoid tasks, continuous control problems via ExoRL, and complex long-horizon planning on OGBench demonstrate the method's versatility across problem domains.

The framework's integration with forward-backward representation learning suggests it complements existing deep learning architectures without requiring architectural modifications. This accessibility could accelerate adoption in industry applications. Future work should examine sample efficiency in high-dimensional state spaces and robustness to out-of-distribution scenarios where new tasks deviate significantly from training data.

• →Successor measures theoretically connect to stationary density ratios, enabling zero-shot policy adaptation without task-specific training
• →The method performs automatic distributional correction by inferring optimal importance sampling weights on-the-fly for any new task
• →Successfully demonstrated across motion tracking, continuous control, and long-horizon planning benchmarks with consistent performance
• →Architecture-agnostic approach integrates seamlessly into existing forward-backward representation frameworks for practical deployment
• →Bridges off-policy learning and zero-shot adaptation paradigms, potentially advancing both research areas and real-world robotics applications