Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning

Researchers propose Group-Graph Policy Optimization (G2PO), a novel reinforcement learning algorithm that transforms linear interaction trajectories into state-transition graphs to improve credit assignment in long-horizon agentic tasks. The method demonstrates significant performance improvements on benchmark tasks like WebShop and ALFWorld, achieving up to 22.2% success rate gains over existing approaches.

G2PO addresses a fundamental challenge in agentic reinforcement learning: how to effectively train AI agents performing multi-step tasks with delayed feedback. Traditional step-level training frameworks treat agent exploration as isolated linear sequences, missing the broader structural patterns that emerge when the same states appear across different interaction paths. By explicitly constructing a global state-transition graph, G2PO leverages this inherent structure to reduce variance in value estimation and provide more targeted credit assignment.

The advancement builds on recent momentum in group-based RL for LLMs, which has shown promise in improving agent behavior through finer-grained training signals. Long-horizon reasoning remains a critical bottleneck for autonomous agents in real-world applications—delayed rewards of dozens of steps create severe credit assignment problems that myopic, trajectory-specific approaches struggle to solve. G2PO's graph-centric perspective represents a meaningful architectural shift in how the field conceptualizes agent learning dynamics.

The benchmark results across WebShop, ALFWorld, and AppWorld demonstrate practical value, with substantial improvements over both prompt-based baselines and reinforcement learning alternatives. These environments reflect realistic agent deployment scenarios requiring multi-turn reasoning. The edge-centric advantage estimation strategy appears particularly effective at identifying critical decision points that drive task completion, suggesting the method captures task structure more effectively than existing approaches.

The framework's success could accelerate adoption of RL methods for agentic applications, potentially influencing how foundation models are fine-tuned for autonomous task execution. Future work likely explores whether graph-based credit assignment scales to even longer horizons and more complex multi-agent scenarios.

• →G2PO transforms linear trajectories into state-transition graphs to improve credit assignment in long-horizon agentic RL tasks
• →Group-aggregation reduces variance by identifying and leveraging identical observations across different interaction paths
• →Edge-centric advantage estimation prioritizes critical state transitions that directly drive task progress
• →Benchmarks show up to 22.2% success rate improvements over GRPO and other state-of-the-art baselines
• →The approach addresses fundamental limitations in step-level training frameworks for delayed-reward environments