Measuring What Matters -- or What's Convenient?: Robustness of LLM-Based Scoring Systems to Construct-Irrelevant Factors

Researchers tested a dual-architecture LLM-based automated scoring system for educational assessments and found it generally robust to construct-irrelevant factors like meaningless text padding and spelling errors. The study shows promise for LLM-based scoring systems' reliability when properly designed, though off-topic responses were heavily penalized.

• →LLM-based scoring systems demonstrated robustness against padding with meaningless text, spelling errors, and writing sophistication variations.
• →Unlike previous non-LLM systems, duplicating large text passages resulted in lower predicted scores on average.
• →Off-topic responses were heavily penalized by the LLM-based scoring system.
• →The dual-architecture approach shows encouraging results for construct-relevant automated assessment design.
• →LLM-based scoring systems may be more resistant to adversarial conditions than traditional automated scoring methods.