Tournament-GRPO: Group-Wise Tournament Rewards for Reinforcement Learning in Open-Ended Long-Form Generation

Researchers propose Tournament-GRPO, a novel reinforcement learning framework that uses group-wise tournament comparisons instead of absolute scoring to improve long-form text generation. By converting rubric-based LLM judgments into relative rewards through competitive rankings, the method achieves 4.52-point improvements over existing approaches on Deep Research Bench benchmarks.

Tournament-GRPO addresses a fundamental challenge in training large language models for open-ended tasks where traditional evaluation metrics fail. The core innovation replaces pointwise LLM-as-a-judge scoring with pairwise comparisons organized as tournaments, enabling the model to learn from relative quality assessments rather than absolute numerical ratings. This approach is particularly valuable because absolute scoring systems struggle with calibration across diverse response types, often fail to discriminate between similar outputs, and plateau during optimization as scores saturate.

The breakthrough builds on growing recognition that relative preference signals are more reliable than absolute scores for LLM training. Prior work in preference-based reinforcement learning established that pairwise comparisons capture meaningful quality distinctions, but Tournament-GRPO extends this by introducing structured group-wise comparisons that accumulate ranking information before normalization. This tournament structure creates richer training signals while maintaining computational efficiency—a critical concern for practitioners scaling RL training.

For the AI research community and LLM developers, this work has immediate implications for improving long-form generation tasks like research synthesis, code generation, and creative writing where reference answers are unavailable. Organizations building search, research, or content-generation systems could leverage these techniques to build more sophisticated reward models without expensive human annotation campaigns. The 4.52-point improvement demonstrates meaningful gains on research quality metrics, suggesting faster convergence to high-performance models during training.

• →Tournament-GRPO replaces absolute LLM scoring with relative group-wise comparisons for more discriminative reward signals
• →Method achieves 4.52-point improvement over baselines on open-ended long-form generation tasks
• →Tournament rewards provide favorable effectiveness-efficiency tradeoffs compared to existing rubric-based approaches
• →Framework addresses saturation and calibration problems inherent in pointwise scoring systems
• →Technique enables improved RL training for tasks without reference answers or automatic metrics