Designing Reliable LLM-Assisted Rubric Scoring for Constructed Responses: Evidence from Physics Exams

Researchers evaluated GPT-4o's ability to score physics exam responses using rubric-assisted scoring, finding that AI reliability matches human inter-rater consistency when rubrics are well-structured and granular. The study reveals that clear rubric design matters far more than LLM configuration choices, with performance declining on ambiguous mid-range responses.

This study addresses a critical gap in educational AI implementation by systematically testing how LLMs can assist with constructed-response scoring in STEM fields. The research moves beyond proof-of-concept demonstrations to examine the practical conditions that determine reliability, comparing GPT-4o performance against human instructors across authentic physics exams with varying response quality and clarity.

The findings emerge from growing pressure in education to scale assessment processes while maintaining consistency. Traditional handwritten STEM responses require significant instructor time and suffer from rater bias, particularly when assigning partial credit. LLMs promise efficiency gains, but implementing them requires understanding which design choices drive reliability. This study systematically isolates these variables—rubric granularity, prompt format, and temperature settings—revealing that structural clarity in rubrics dominates other factors.

The practical implications extend beyond physics education. The research demonstrates that AI-assisted scoring performs comparably to human reliability on high-confidence cases but struggles with ambiguous middle-ground responses. This pattern suggests limitations in how current LLMs handle nuanced judgment calls requiring contextual reasoning. For educational institutions and assessment platforms, the findings indicate that investing in rubric design yields better returns than fine-tuning LLM parameters.

The work establishes a replicable methodology for evaluating AI scoring across other STEM disciplines and subjects. Future implementations should prioritize checklist-based rubrics over holistic scoring frameworks and recognize that temperature adjustments provide minimal benefit compared to prompt clarity. The research provides a template for responsible AI adoption in high-stakes assessment contexts where transparency and consistency directly impact student outcomes.

• →Rubric design quality matters more than LLM configuration choices for reliable AI-assisted scoring
• →Fine-grained, checklist-based rubrics significantly outperform holistic scoring approaches
• →AI reliability matches human inter-rater agreement on high and low-performing responses but declines on ambiguous mid-range cases
• →Temperature settings have minimal impact on scoring consistency compared to prompt formatting
• →Clear, skill-based rubrics enable transferable AI scoring implementation across STEM assessment contexts