Brittlebench: Quantifying LLM robustness via prompt sensitivity

Researchers introduce Brittlebench, a new evaluation framework that reveals frontier AI models experience up to 12% performance degradation when faced with minor prompt variations like typos or rephrasing. The study shows that semantics-preserving input perturbations can account for up to half of a model's performance variance, highlighting significant robustness issues in current language models.

• →Current AI evaluation methods using clean benchmarks overestimate model performance and fail to capture real-world input variability.
• →Brittlebench testing reveals frontier models suffer up to 12% performance degradation from minor prompt variations.
• →A single prompt perturbation changes relative model rankings in 63% of test cases, affecting comparative performance conclusions.
• →Semantics-preserving input variations can account for up to 50% of performance variance in state-of-the-art models.
• →The framework demonstrates critical need for more robust AI model evaluation and development approaches.