143 articles tagged with #policy-optimization. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers propose Posterior Sampling-based Policy Optimization (PSPO), a novel approach to offline reinforcement learning that addresses the critical challenge of balancing model generalization with robustness against exploitation errors. By formulating dynamics modeling as Bayesian inference, PSPO enables safer learning from out-of-distribution data while maintaining theoretical convergence guarantees.
Researchers introduce Model-Driven Policy Optimization (MDPO), a framework that enhances gradient-based optimization in differentiable simulators by incorporating adaptive stochastic exploration. The method dynamically adjusts noise injection based on gradient sensitivity, enabling better navigation of complex optimization landscapes and outperforming both deterministic planning and model-free reinforcement learning approaches on nonlinear benchmark tasks.
Researchers propose GXPO, a new policy optimization technique for reinforcement learning that approximates multi-step lookahead using only three backward passes instead of many, improving large language model reasoning performance by 1.65-5.00 points over standard GRPO while achieving up to 4x step speedup.
Researchers propose RRCM, a novel framework that enhances Large Language Model-based recommendation systems by dynamically retrieving relevant collaborative and metadata information. The system learns optimal context construction through ranking-driven optimization, addressing key challenges in balancing context quality with efficiency limitations.
Researchers propose CTPO (Cumulative Token Policy Optimization), a new approach to reinforcement learning for large language models that addresses the bias-variance tradeoff in importance sampling ratios. By using cumulative token-level ratios with position-adaptive clipping, CTPO achieves superior performance on mathematical reasoning benchmarks compared to existing methods like PPO and GRPO.
Researchers introduce POISE, a reinforcement learning method that uses a language model's internal hidden states to estimate baseline values for policy optimization, eliminating the computational overhead of separate critic models. The approach demonstrates comparable performance to existing methods while requiring significantly less compute, enabling more efficient training of large reasoning models.
Researchers introduce Drifting Field Policy (DFP), a one-step generative policy that uses Wasserstein gradient flow to optimize reinforcement learning without ODE-based approaches. DFP demonstrates state-of-the-art performance on robotic manipulation tasks, suggesting a potential shift in how generative models are applied to control problems.
Researchers introduce Safety-Biased Trust Region Policy Optimisation (SB-TRPO), a reinforcement learning algorithm designed to satisfy strict safety constraints in critical applications while maintaining task performance. The method dynamically balances safety compliance with reward improvement through principled policy updates, with formal guarantees of safety progress.
Skill1 presents a unified reinforcement learning framework that enables language model agents to co-evolve three coupled capabilities: skill selection, utilization, and distillation from a single task-outcome reward signal. Demonstrated improvements over existing baselines on complex tasks suggest advances in how AI agents can build and leverage persistent skill libraries across diverse problem domains.
Researchers propose Listwise Policy Optimization (LPO), a new framework for training large language models that improves upon existing reinforcement learning approaches by explicitly projecting policies toward target distributions on the response simplex. The method demonstrates consistent performance improvements across reasoning tasks while maintaining training stability and response diversity.
Researchers introduce Maximum Entropy Adjoint Matching (ME-AM), a new framework for offline reinforcement learning that combines flow-matching generative policies with entropy regularization to overcome limitations in existing Q-learning approaches. The method addresses popularity bias and support binding issues that prevent agents from discovering high-reward actions in low-density regions, demonstrating competitive performance across continuous control benchmarks.
Researchers propose CL-MARL, a curriculum learning framework for multi-agent reinforcement learning that dynamically adjusts task difficulty based on agent performance, addressing a fundamental limitation where fixed-difficulty training constrains policy generalization. The method achieves 40% win rate on complex cooperative tasks, outperforming existing baselines by significant margins.
PORTool is a new policy-optimization algorithm that improves how AI agents learn to use external tools by solving the credit-assignment problem in multi-step reasoning tasks. The method uses a rewarded tree structure to assign rewards at individual steps rather than only at outcomes, enabling agents to achieve higher accuracy while reducing unnecessary tool calls.
RHyVE is a new verification and deployment protocol for LLM-generated reward functions in reinforcement learning that addresses a critical gap: when and how to use AI-generated rewards during policy training. The research demonstrates that reward reliability depends on policy competence levels and training phases, requiring adaptive deployment strategies rather than static scheduling.
Researchers introduce EXPO, a reinforcement learning algorithm that trains expressive policies (like diffusion models) more efficiently by avoiding direct value optimization. The method uses a lightweight Gaussian policy to edit actions from a base policy, achieving 2-3x improvements in sample efficiency for both offline-to-online and fine-tuning scenarios.
Researchers introduce TeLAPA, a continual reinforcement learning framework that maintains diverse policy archives instead of relying on single-model preservation, addressing the loss of plasticity problem where retained policies fail to serve as effective starting points for rapid adaptation across new tasks.
Researchers demonstrate that reward-weighted classifier-free guidance (RCFG) can dynamically adjust autoregressive model outputs to optimize arbitrary reward functions at test time without retraining. Applied to molecular generation, this approach enables real-time optimization of competing objectives and accelerates reinforcement learning convergence when used as a teacher for policy distillation.
Researchers propose Adaptive Entropy Regularization (AER), a dynamic framework that addresses policy entropy collapse in LLM reinforcement learning by adjusting exploration intensity based on task difficulty. The method improves upon fixed entropy regularization approaches, demonstrating consistent gains in mathematical reasoning benchmarks while maintaining balanced exploration-exploitation tradeoffs.
Researchers present a theoretical framework comparing entropy control methods in reinforcement learning for LLMs, showing that covariance-based regularization outperforms traditional entropy regularization by avoiding policy bias and achieving asymptotic unbiasedness. This analysis addresses a critical scaling challenge in RL-based LLM training where rapid policy entropy collapse limits model performance.
Researchers propose Policy Split, a novel reinforcement learning approach for LLMs that uses dual-mode entropy regularization to balance exploration with task accuracy. By bifurcating policy into normal and high-entropy modes, the method enables diverse behavioral patterns while maintaining performance, showing improvements over existing entropy-guided RL baselines.
Researchers introduce PODS (Policy Optimization with Down-Sampling), a technique that accelerates reinforcement learning training for large language models by selectively training on high-variance rollouts rather than all generated data. The method achieves equivalent performance to standard approaches at 1.7x faster speeds, addressing computational bottlenecks in LLM reasoning optimization.
Researchers propose StaRPO, a reinforcement learning framework that improves large language model reasoning by incorporating stability metrics alongside task rewards. The method uses Autocorrelation Function and Path Efficiency measurements to evaluate logical coherence and goal-directedness, demonstrating improved accuracy and reasoning consistency across four benchmarks.
Researchers propose Visually-Guided Policy Optimization (VGPO), a framework that enhances vision-language models' ability to focus on visual information during reasoning tasks. The method addresses a fundamental limitation where text-dominated VLMs suffer from weak visual attention and temporal visual forgetting, improving performance on multimodal reasoning and visual-dependent tasks.
Researchers propose T-STAR, a novel reinforcement learning framework that structures multi-step agent trajectories as trees rather than independent chains, enabling better credit assignment for LLM agents. The method uses tree-based reward propagation and surgical policy optimization to improve reasoning performance across embodied, interactive, and planning tasks.
Researchers propose MA-VLCM, a framework that uses pretrained vision-language models as centralized critics in multi-agent reinforcement learning instead of learning critics from scratch. This approach significantly improves sample efficiency and enables zero-shot generalization while producing compact policies suitable for resource-constrained robots.
