EvoRubrics: Dynamic Rubrics as Rewards via Adversarial Co-Evolution for LLM Reinforcement Learning

EvoRubrics introduces a co-evolutionary reinforcement learning framework where a Policy LLM and Rubric Generator jointly improve through adversarial interaction, addressing the limitation of static reward criteria that lose discriminative power as models improve. The approach enables real-time evaluation adaptation and generates transferable reward models, with experiments showing consistent improvements over static and dynamic baselines.

EvoRubrics represents a significant advancement in reinforcement learning for large language models by tackling a fundamental problem in open-ended task optimization: the degradation of reward signals as models improve. Traditional rubric-based rewards rely on fixed criteria that become less informative over time, leading to reward saturation and potential gaming of the reward function. This research introduces a dynamic solution through adversarial co-evolution, where evaluation mechanisms adapt continuously alongside policy improvements.

The framework addresses limitations of previous dynamic rubric approaches that depend on external frontier models or ground-truth answers. By embedding the rubric generation process within the training loop, EvoRubrics enables fine-grained, continuous adaptation rather than coarse periodic updates. This creates a natural curriculum effect where evaluation difficulty scales with model capability, maintaining consistent learning signal quality throughout training.

The practical implications extend beyond theoretical improvements in training efficiency. The learned Rubric Generator demonstrates transferability as a reward model, suggesting potential applications across different LLM training scenarios. The fully self-supervised variant achieving meaningful gains without external supervision indicates that co-evolution between generation and evaluation provides sufficient learning signals independent of human-annotated data, reducing dependency on expensive labeling infrastructure.

For AI researchers and practitioners, this approach opens pathways for more scalable and self-improving training systems. The publicly available implementation enables rapid adoption and validation across different domains. Looking ahead, the generalizability of evolved rubrics and their application to multi-agent learning systems merit investigation, as does the scalability of this approach to larger model scales and more complex task distributions.

• →EvoRubrics enables dynamic reward criteria that adapt in real-time as policy improves, preventing reward saturation and gaming behavior.
• →The co-evolutionary framework achieves consistent performance gains over static and dynamic baselines without requiring external frontier models or ground-truth supervision.
• →Learned Rubric Generators demonstrate transferability as reward models across different tasks, reducing reliance on task-specific human annotation.
• →Self-supervised variant succeeds without external supervision, suggesting co-evolution between generation and evaluation provides sufficient learning signals.
• →Automatic curriculum emerges naturally from the adversarial interaction between policy and rubric generator, improving training efficiency.