Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMs

Researchers introduce Owen-Shapley Policy Optimization (OSPO), a reinforcement learning algorithm that improves how language models learn from feedback by attributing credit to individual tokens rather than treating entire sequences as atomic units. The method addresses a fundamental training gap in generative AI systems used for recommendation tasks, showing measurable improvements on real e-commerce datasets.

OSPO addresses a critical inefficiency in current large language model training pipelines. Standard reinforcement learning approaches like GRPO provide only sequence-level rewards, making it impossible for models to understand which specific tokens or phrases drove successful outcomes. This credit assignment problem becomes acute when models must infer latent user preferences from ambiguous language—a reasoning capability rarely developed during pretraining but essential during deployment. The Owen-Shapley framework leverages game theory principles to decompose sequence-level advantages into token-level contributions, effectively creating a finer-grained reward signal without requiring additional value function networks.

This research emerges from the broader push to scale language models beyond next-token prediction into agentic reasoning and personalization domains. As companies deploy LLMs for recommendation systems and search tasks, training efficiency directly impacts model quality and computational costs. The method's reliance on Shapley-Owen attributions—decomposing contributions based on semantic coalitions like phrases or sentences—preserves training stability while improving interpretability.

For the AI industry, OSPO represents incremental but meaningful progress in RL training efficiency. Companies developing recommendation systems could achieve better model performance with similar computational budgets. The controlled experiments on Amazon ESCI and H&M Fashion datasets demonstrate practical applicability, while robustness gains against out-of-distribution retrievers suggest real-world deployment benefits.

The significance lies not in revolutionary capability gains but in making RL-trained generative models more efficient and interpretable. As LLM training becomes increasingly expensive, algorithmic improvements that squeeze better performance from existing compute budgets drive long-term competitive advantages in AI development.

• →OSPO solves the credit assignment problem in LLM training by attributing sequence-level rewards to individual tokens and semantic units
• →The method eliminates the need for parametric value models, reducing training complexity while preserving policy optimality
• →Experiments show consistent improvements on e-commerce recommendation tasks with enhanced robustness to distribution shift
• →The framework enables better interpretability of which model behaviors drive high-quality outputs in reasoning-heavy tasks
• →Shapley-Owen attributions provide a theoretically grounded alternative to sparse reward signals in generative AI training