Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

Researchers propose Faithful Agentic XAI (FAX), a framework that improves the reliability of AI explanations generated by large language models through explicit verification mechanisms. The study introduces CRAFTER-XAI-Bench, a new benchmark for testing explanation faithfulness in complex environments, demonstrating that current XAI systems can produce plausible but inaccurate explanations that mislead users.

The emergence of agentic AI systems powered by large language models has created a critical problem: explanations that sound convincing but lack grounding in actual model behavior. FAX addresses this by implementing a verification layer that decomposes explanations into verifiable claims and cross-references them against reliable tools before presenting final outputs. This approach recognizes that LLMs excel at generating fluent, natural language but struggle with factual consistency when applied to complex model interpretations.

The research builds on growing recognition that explainability must go beyond surface-level clarity. As AI systems increasingly influence high-stakes decisions in finance, healthcare, and other domains, users need explanations they can trust. Previous XAI benchmarks often conflated task accuracy with true model-specific faithfulness, creating a false sense of reliability. CRAFTER-XAI-Bench corrects this by testing explanations specifically against target model behavior in reinforcement learning environments with diverse, complex scenarios.

The performance improvement from 0.20 to 0.46 in simulation faithfulness demonstrates substantial room for progress in current systems. This matters for enterprises deploying AI systems where understanding failure modes is critical. The framework suggests that human-AI interaction design requires explicit verification rather than relying on LLM-generated explanations at face value. For developers and researchers, the findings underscore that benchmarking XAI systems demands scenario complexity that captures real-world model behavior rather than simplified tabular tasks.

• →FAX framework improves explanation faithfulness to 0.46 from 0.20 baseline through explicit verification against faithful tools
• →Current XAI benchmarks conflate task accuracy with model-specific faithfulness, failing to properly assess explanation reliability
• →LLMs can amplify unreliable explanations by generating plausible but unfaithful interpretations of complex model behavior
• →CRAFTER-XAI-Bench provides an open-world reinforcement learning benchmark for more rigorous XAI evaluation
• →Verification mechanisms are essential for trustworthy agentic AI systems in high-stakes applications