IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning

Researchers propose Imaginary Planning Distillation (IPD), a novel framework that enhances offline reinforcement learning by incorporating planning into sequential policy models. IPD uses world models and Model Predictive Control to generate optimal rollouts, training Transformer-based policies that significantly outperform existing methods on D4RL benchmarks.

• →IPD addresses limitations of decision transformer-based sequential policies in offline reinforcement learning through integrated planning.
• →The framework combines world models with uncertainty measures and quasi-optimal value functions to identify and improve suboptimal trajectories.
• →Model Predictive Control generates reliable imagined optimal rollouts to augment training datasets.
• →Transformer-based sequential policies trained with IPD show significant performance improvements over state-of-the-art methods.
• →Empirical evaluations on D4RL benchmark demonstrate superior results across diverse reinforcement learning tasks.