Reinforcement Learning with Robust Rubric Rewards

Researchers introduce RLR³, an advanced reinforcement learning framework that extends reward verification from task-level to criterion-level evaluation, enabling multi-criteria supervision for vision-language tasks. The approach uses hybrid verification paths combining LLM extractors with deterministic verifiers or LLM judges, demonstrating a 4.7-point improvement over baseline models on 15 benchmarks.

RLR³ addresses a fundamental challenge in reinforcement learning systems: how to provide reliable, granular feedback for complex vision-language tasks that resist simple binary verification. Traditional RLVR approaches struggle with partially verifiable tasks requiring multiple evaluation dimensions—perceptual quality, reasoning correctness, and constraint satisfaction. The proposed framework elegantly handles this by implementing criterion-level verification, routing different aspects of task evaluation through specialized execution paths depending on their verifiability characteristics.

The research builds on the growing trend toward more sophisticated reward mechanisms in AI training. As vision-language models become more capable, simple pass-fail reward signals prove insufficient. Rubric-based evaluation mirrors human assessment practices, where multiple weighted criteria determine overall quality. The minimal exposure strategy—masking ground truths from extractors and images from judges—directly tackles a critical vulnerability: models gaming evaluation systems through spurious correlations rather than achieving genuine task performance.

The technical innovation of hierarchical aggregation prioritizing essential criteria reflects practical deployment concerns. In real-world applications, some evaluation dimensions matter substantially more than others; the system's ability to weight these appropriately has direct implications for model behavior. The 4.7-point improvement over baseline models and the demonstrated superiority to official instruct-to-thinking benchmarks suggest substantial practical value. For organizations developing vision-language systems, this work provides a methodologically rigorous approach to improving model quality through better training feedback.

• →RLR³ extends reward verification to criterion-level evaluation, enabling nuanced supervision for partially verifiable vision-language tasks.
• →The minimal exposure strategy reduces exploitable false positives by preventing models from gaming evaluation metrics through spurious correlations.
• →Hierarchical aggregation prioritizes essential evaluation criteria, reflecting real-world deployment requirements where certain dimensions carry greater importance.
• →Evaluation across 15 benchmarks shows 4.7-point improvement over baseline models, outperforming official instruct-to-thinking variants.
• →Hybrid verification combining deterministic verifiers and LLM judges provides flexibility for tasks spanning verifiable and non-verifiable criteria.