Researchers propose a computational model that evaluates explanations by converting them into executable action plans through large language models and planning agents. Across four experiments with 1,200 explanations, higher-scored explanations correlate with improved navigation performance and user helpfulness judgments, demonstrating that explanation quality can be measured by practical outcomes under uncertainty.
This research addresses a fundamental challenge in human-AI interaction: how to measure whether an explanation actually helps someone accomplish a goal. Traditional evaluation methods rely on subjective judgments or linguistic metrics, but this work anchors explanation quality to concrete behavioral outcomes—how successfully a person can navigate based on given instructions.
The methodology is novel and rigorous. By converting natural language explanations into formal policy priors and value maps that guide a planning agent navigating partial observability scenarios, the researchers created an objective measurement framework. The preregistered experimental design with 1,200 explanations across 24 different maps reduces bias and strengthens reproducibility. Critically, their scoring mechanism penalizes replanning, capturing real-world constraints where good explanations minimize course corrections.
The results validate an important principle: explanation effectiveness is not abstract but measurable through utility. Participants with high-scoring explanations significantly outperformed both those without explanations and those given low-scoring ones. This finding has practical implications for AI systems that generate user guidance, from navigation apps to customer support chatbots to educational software.
For the AI industry, this research provides a quantifiable approach to improving language model outputs in task-oriented contexts. Rather than optimizing for fluency or grammatical correctness, systems can be trained toward explanations that minimize user uncertainty and required replanning. The work bridges cognitive science, formal planning theory, and large language models—demonstrating how explanation quality should be grounded in observable action outcomes rather than linguistic properties alone.
- →Explanation quality can be objectively measured by converting language into executable action plans and scoring path efficiency and reliability.
- →High-scoring explanations significantly improve real-world navigation performance compared to low-scoring or absent explanations.
- →The model captures how people mentally simulate instructions before acting, enabling evaluation of communication effectiveness under uncertainty.
- →Replanning penalties in the scoring mechanism reward clear explanations that reduce user confusion and course corrections.
- →This framework applies broadly to any task-oriented AI system generating guidance, from navigation to customer support to education.