RubricBench: Aligning Model-Generated Rubrics with Human Standards

RubricBench is a new benchmark with 1,147 pairwise comparisons designed to evaluate rubric-based assessment methods for Large Language Models. Research reveals a significant gap between human-annotated and AI-generated rubrics, showing that current state-of-the-art models struggle to autonomously create valid evaluation criteria.

• →RubricBench introduces the first unified benchmark for assessing rubric-based evaluation of Large Language Models.
• →The benchmark contains 1,147 carefully curated pairwise comparisons with expert-annotated atomic rubrics.
• →Current state-of-the-art AI models show substantial capability gaps compared to human-generated evaluation rubrics.
• →The research targets complex samples with nuanced input and misleading surface biases to improve evaluation reliability.
• →Model-generated rubrics lag considerably behind human-guided performance in creating valid evaluation criteria.