Do Metrics for Counterfactual Explanations Align with User Perception?

A new study reveals that standard algorithmic metrics used to evaluate AI counterfactual explanations poorly correlate with human perceptions of explanation quality. The research found weak and dataset-dependent relationships between technical metrics and user judgments, highlighting fundamental limitations in current AI explainability evaluation methods.

• →Current algorithmic metrics for evaluating counterfactual AI explanations show weak correlations with human quality assessments.
• →The relationship between technical metrics and user perceptions varies significantly across different datasets.
• →Combining multiple evaluation metrics does not improve the ability to predict human judgments of explanation quality.
• →The findings reveal structural limitations in how current metrics capture human-relevant criteria for AI explanations.
• →The study calls for more human-centered approaches to evaluating explainable AI systems.