A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness

A new research study reveals that AI judges used to evaluate the safety of large language models perform poorly when assessing adversarial attacks, often degrading to near-random accuracy. The research analyzed 6,642 human-verified labels and found that many attacks artificially inflate their success rates by exploiting judge weaknesses rather than generating genuinely harmful content.

• →LLM-as-a-Judge frameworks show severe performance degradation when evaluating adversarial attacks on AI safety.
• →Judge performance often drops to near-random chance due to distribution shifts in red-teaming scenarios.
• →Many reported attack success rates are artificially inflated by exploiting judge insufficiencies rather than generating truly harmful content.
• →The study introduces ReliableBench and JudgeStressTest datasets to enable more accurate AI safety evaluation.
• →Current validation protocols fail to account for the diverse generation styles and output patterns in adversarial scenarios.