Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parameteric Policies

Researchers present theoretical advances in offline reinforcement learning that extend beyond current limitations to work with parameterized policies over large or continuous action spaces. The work connects mirror descent to natural policy gradient methods and reveals a surprising unification between offline RL and imitation learning.

• →Current offline RL algorithms like PSPI are limited to finite and small action spaces, restricting practical applications.
• →The research extends theoretical guarantees to parameterized policy classes over large or continuous action spaces.
• →Contextual coupling is identified as the core difficulty when extending mirror descent to parameterized policies.
• →The work reveals a novel unification between offline reinforcement learning and imitation learning through natural policy gradients.
• →The approach accommodates standalone policy parameterization which is commonly used in practice but not well-supported theoretically.