Robust LLM Performance Certification via Constrained Maximum Likelihood Estimation

Researchers propose a new constrained maximum likelihood estimation (MLE) method to accurately estimate failure rates of large language models by combining human-labeled data, automated judge annotations, and domain-specific constraints. The approach outperforms existing methods like Prediction-Powered Inference across various experimental conditions, providing a more reliable framework for LLM safety certification.

• →New constrained MLE method integrates human labels, automated annotations, and domain constraints for better LLM failure rate estimation.
• →The approach consistently delivers more accurate and lower-variance estimates than state-of-the-art baselines like Prediction-Powered Inference.
• →Method addresses the current tradeoff between expensive human gold standards and biased automatic annotation schemes.
• →Framework moves beyond black-box automated judges to provide a principled and interpretable solution.
• →Research provides a scalable pathway towards safer LLM deployment through rigorous failure rate certification.