Reinforcement Learning with Pairwise Preferences in Long-Term Decision Problems

Researchers propose 'Markov decision contests' as a new reinforcement learning framework that leverages pairwise preferences instead of scalar rewards, proving that stationary Markov policies are optimal and demonstrating superior learning efficiency in long-horizon problems compared to existing methods.

This research addresses a fundamental challenge in reinforcement learning: the difficulty of specifying reward functions at scale. While traditional RL frameworks require scalar reward definitions, pairwise preferences—comparing outcomes A versus B—often align better with how humans naturally express objectives. The proposed Markov decision contest framework bridges theory and practice by establishing that simpler, stationary Markov policies outperform more complex history-dependent alternatives, a critical theoretical guarantee that reduces computational complexity.

The work builds on growing recognition that preference-based learning better captures nuanced goals that scalar rewards struggle to represent. Prior approaches to preference-based RL suffered from inefficiency in long-horizon problems, limiting their practical application. By proving the problem is solvable in polynomial time and demonstrating sublinear convergence rates, the authors provide theoretical foundations previously absent from this domain.

The practical impact emerges through empirical testing on high-dimensional decision problems with extended time horizons. The authors show their approximate algorithm significantly outperforms existing preference-based methods, suggesting real deployment advantages. This matters for domains where reward engineering remains expensive: autonomous systems, recommendation algorithms, and complex control tasks where human feedback through pairwise comparisons is more reliable than numerical scores.

Looking forward, the framework's efficiency gains could accelerate preference-based RL adoption in production systems. The theoretical guarantees around Markov policy optimality reduce uncertainty in system design, while the polynomial-time solvability suggests scalability. Extensions to multi-agent scenarios and integration with large language models using preference feedback represent natural next steps.

• →Markov decision contests prove stationary policies are optimal in preference-based reinforcement learning, simplifying algorithm design.
• →The framework achieves polynomial-time solvability for exact solutions, providing theoretical foundations absent from prior work.
• →Approximate algorithms demonstrate significantly better learning efficiency than existing methods on long-horizon, high-dimensional problems.
• →Pairwise preferences offer a more natural specification mechanism than scalar rewards for complex, nuanced objectives.
• →The work bridges theory-practice gap by guaranteeing Markov policies outperform history-dependent alternatives.