LongTraceRL: Learning Long-Context Reasoning from Search Agent Trajectories with Rubric Rewards

Researchers introduce LongTraceRL, a reinforcement learning method that improves large language models' ability to reason over lengthy documents by using search agent trajectories and entity-level reward signals. The approach generates challenging training contexts with high-confusability distractors and applies rubric rewards that supervise intermediate reasoning steps, demonstrating consistent improvements across multiple LLM sizes and benchmarks.

LongTraceRL addresses a fundamental limitation in large language models: their struggle to locate and synthesize relevant information when processing long contexts filled with distracting content. The research tackles this through two innovations in training methodology. First, the data construction leverages actual search agent behavior to create more realistic and challenging distractors—documents the agent examined but rejected, plus results never opened. This approach produces training scenarios significantly harder than random sampling or single-search methods. Second, the rubric reward system moves beyond binary outcome signals by providing fine-grained supervision at the entity level, tracking whether the model identifies correct entities along reasoning chains.

This work builds on the growing recognition that reinforcement learning with verifiable rewards can enhance reasoning capabilities in language models. Previous attempts faltered due to weak distractors and sparse reward signals that couldn't guide intermediate steps. LongTraceRL's positive-only reward strategy prevents reward hacking while differentiating reasoning quality among correct answers, addressing practical challenges in RL-based model training.

The research demonstrates broad applicability across model sizes (4B to 30B parameters) and multiple long-context benchmarks, suggesting the method generalizes effectively. For the AI research community, this represents progress toward more reliable reasoning systems capable of handling document-intensive tasks. The public release of code, datasets, and models enables reproducibility and adoption. The implications extend to applications requiring complex information retrieval and synthesis, from research assistance to fact-checking systems.

• →LongTraceRL uses search agent trajectories to create high-confusability distractors, making training contexts substantially more challenging than conventional methods.
• →Rubric rewards provide entity-level supervision along reasoning chains, moving beyond outcome-only signals to guide intermediate reasoning steps.
• →The method shows consistent improvements across models ranging from 4B to 30B parameters, demonstrating scalability and generalization.
• →Positive-only reward strategy prevents reward hacking while enabling fine-grained differentiation of reasoning quality among correct responses.
• →Publicly released code and datasets enable community adoption and further research in long-context reasoning for language models.