Reason in Chains, Learn in Trees: Self-Rectification and Grafting for Multi-turn Agent Policy Optimization

Researchers propose T-STAR, a novel reinforcement learning framework that structures multi-step agent trajectories as trees rather than independent chains, enabling better credit assignment for LLM agents. The method uses tree-based reward propagation and surgical policy optimization to improve reasoning performance across embodied, interactive, and planning tasks.

This research addresses a fundamental challenge in reinforcement learning for large language models: the difficulty of assigning credit accurately in multi-step reasoning tasks where sparse rewards make it hard to identify which intermediate steps drive success or failure. Traditional approaches treat trajectories as independent sequences, missing correlations that could improve learning efficiency. T-STAR's innovation lies in consolidating seemingly disparate trajectories into a unified cognitive tree structure by identifying functionally equivalent steps, then back-propagating rewards through this tree to generate step-level advantage estimates. This architectural shift enables more granular understanding of reasoning processes. The framework also introduces In-Context Thought Grafting, which synthesizes corrective reasoning by contrasting successful and failed branches at critical divergence points—essentially learning from contrastive examples automatically discovered during training. The surgical policy optimization component concentrates gradient updates on these identified critical steps rather than distributing updates uniformly across entire chains. For the AI development community, this represents progress toward more sample-efficient training of reasoning agents, which directly impacts practical deployment feasibility and cost. The breadth of benchmark improvements across embodied, interactive, reasoning, and planning domains suggests the approach generalizes meaningfully rather than overfitting to specific task types. As language models increasingly power autonomous agents in real-world applications, more efficient credit assignment mechanisms reduce training compute requirements and accelerate development cycles. This work contributes valuable techniques for practitioners building production LLM agents, though the research remains in academic publication phase without immediate market implications.

• →T-STAR consolidates independent trajectories into tree structures to recover hidden reward correlations missed by standard approaches.
• →Introspective Valuation mechanism enables variance-reduced step-level advantage estimation through reward back-propagation in cognitive trees.
• →In-Context Thought Grafting synthesizes corrections by contrasting successful and failed branches at critical decision points.
• →Surgical Policy Optimization concentrates gradient updates on identified critical steps rather than uniformly across chains.
• →Consistent improvements demonstrated across embodied, interactive, reasoning, and planning benchmarks with largest gains on extended reasoning tasks.