No Single Metric Tells the Whole Story: A Multi-Dimensional Evaluation Framework for Uncertainty Attributions

Researchers propose a new framework for evaluating uncertainty attribution methods in explainable AI, addressing inconsistent evaluation practices in the field. The study introduces five key properties including a new 'conveyance' metric and demonstrates that gradient-based methods outperform perturbation-based approaches across multiple evaluation criteria.

• →Current evaluation of uncertainty attribution methods in XAI lacks standardization and relies on inconsistent metrics.
• →The proposed framework aligns uncertainty attributions with the established Co-12 framework using five properties: correctness, consistency, continuity, compactness, and conveyance.
• →Gradient-based methods consistently outperform perturbation-based approaches in consistency and conveyance metrics.
• →Monte-Carlo dropconnect demonstrates superior performance compared to Monte-Carlo dropout across most evaluation metrics.
• →No single metric adequately evaluates uncertainty attribution quality, requiring multi-dimensional assessment approaches.