Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges

Researchers developed a curriculum-based training method for safety judges that dramatically improves their consistency across different evaluation rubrics. The approach combines dynamic rubric generation with a staged learning process, achieving 94.12-94.88% accuracy with minimal variance across three different rubric styles, outperforming larger general-purpose and specialized LLMs.

Safety judges—AI systems tasked with evaluating whether model outputs comply with safety guidelines—have emerged as critical infrastructure for responsible AI deployment. However, recent research revealed these judges suffer from significant brittleness, with accuracy swings of up to 24 percentage points when evaluation criteria are reformatted stylistically. This fragility undermines confidence in automated safety systems, particularly as organizations deploy different safety frameworks across teams and domains.

The core insight here is treating safety judgment as a rubric-following problem rather than a classification task. Rather than memorizing specific evaluation templates, a robust judge must understand the underlying principles and apply them consistently regardless of how the rubric is formulated. The researchers' solution—combining dynamically generated rubrics with a curriculum that progresses from clean, fixed-rubric data to noisier variable-rubric data—mirrors how humans learn to apply abstract principles across contexts.

The results demonstrate meaningful progress. A 12-billion parameter model trained with this curriculum achieves near-state-of-the-art accuracy while maintaining cross-rubric consistency that 30-billion parameter general models cannot match. The ablation study proves the curriculum scheduling is essential; naive mixing of dynamic rubrics without careful curriculum design actually increases variance threefold.

This work matters for AI safety infrastructure and large model deployment. Organizations implementing automated safety evaluation need judges that maintain consistent standards as evaluation frameworks evolve or differ across teams. The research also suggests broader applications for curriculum learning in making AI systems more robust to natural distribution shifts. Future work should test whether these techniques transfer across entirely different safety domains and evaluate performance under adversarial rubric manipulation.

• →Curriculum-based training combining fixed and dynamic rubrics reduces cross-rubric variance to 0.76 compared to 3.60 for naive mixing approaches.
• →A 12B safety judge outperforms dedicated 30B classifiers and general LLMs in both accuracy consistency and peak performance across multiple rubric formats.
• →Safety judgment requires learning abstract evaluation principles rather than memorizing specific rubric templates to remain robust under formulation changes.
• →Naive fine-tuning on variable rubrics degrades model stability; progressive curriculum scheduling from clean to noisy data proves essential for robustness.
• →The approach generalizes across three distinct rubric styles (HarmBench, ShieldGemma, and domain-specific), indicating applicability to diverse safety evaluation scenarios.