Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Researchers introduce Pass@K Policy Optimization (PKPO), a reinforcement learning method that optimizes for multiple solution attempts jointly rather than individually, enabling better exploration and problem-solving on harder tasks. The approach derives unbiased estimators for pass@k performance across arbitrary k values and demonstrates improved learning on challenging benchmarks using open-source LLMs.

Pass@K Policy Optimization addresses a fundamental inefficiency in how reinforcement learning algorithms handle multiple sampling attempts. Traditional RL methods optimize pass@1 performance, treating each solution attempt in isolation and neglecting how samples might collectively solve problems. PKPO reframes the reward structure to consider sets of samples as units, prioritizing diversity and combined utility over individual sample strength.

This research builds on growing recognition that exploration capacity in RL remains underutilized for complex problems. Previous attempts to optimize pass@k were constrained to k=n (all samples), creating a false choice between pass@1 and pass@k gains. PKPO's innovation lies in enabling flexible optimization of arbitrary k values with low-variance, unbiased gradient estimators that reduce to standard RL with transformed rewards. The ability to anneal k during training means models can optimize both metrics simultaneously rather than trading one for another.

For the AI development community, PKPO has significant implications for scaling language model reasoning. The real-world validation using GEMMA-2 demonstrates practical applicability beyond toy problems. By unblocking learning on challenging task sets where conventional optimization stalls, the method suggests that ensemble-like approaches during training can substantially improve problem-solving capabilities. This approach mirrors how best-of-N sampling works at inference time but optimizes the underlying model toward those capabilities.

Developers building reasoning-intensive AI systems should monitor implementations of PKPO, as it directly addresses the exploration-exploitation tradeoff in complex problem domains. The method's efficiency gains could reduce training costs while improving final model performance on difficult benchmarks, potentially accelerating progress in AI reasoning and code generation tasks.

• →PKPO optimizes reinforcement learning by considering multiple solution attempts jointly rather than individually, improving exploration on harder problems.
• →The method derives unbiased estimators for pass@k performance that work with arbitrary k values, not just k=n as in previous approaches.
• →Training can simultaneously optimize both pass@1 and pass@k metrics through k-annealing, eliminating the traditional performance tradeoff.
• →Real-world validation on GEMMA-2 shows the approach unblocks learning on challenging task sets where standard pass@1 optimization fails.
• →The transformation reduces to standard RL with modified rewards, making implementation feasible without major algorithmic changes.