2-Step Agent: A Framework for the Interaction of a Decision Maker with AI Decision Support

Researchers present the 2-Step Agent framework to model how decision makers learn from ML-based decision support systems. The study reveals that even when ML models are well-specified and agents behave rationally, misaligned prior beliefs can cause ML-DS to produce worse outcomes than no support at all, highlighting critical risks in deploying AI for high-stakes decisions.

The 2-Step Agent framework addresses a fundamental gap in understanding human-AI collaboration in decision-critical domains like healthcare and law. Rather than treating ML predictions as static outputs, the research models them as information carriers that influence decision makers' beliefs through Bayesian inference. This distinction matters because predictions embed patterns from training data, allowing agents to update their understanding of underlying causes and effects.

The framework's most significant finding challenges the assumption that better ML models automatically improve outcomes. The researchers demonstrate that a single misaligned prior belief—a decision maker's incorrect assumption about how the ML model works or what it measures—can render decision support counterproductive even with perfect models and rational agents. This reveals a critical failure mode in real-world deployments where stakeholders often hold incorrect mental models about AI systems.

For practitioners deploying AI in consequential domains, this research suggests that model accuracy alone is insufficient. Decision makers must develop accurate mental models of the system's capabilities, limitations, and assumptions. The tractable solutions derived for linear Gaussian settings provide a foundation for analyzing specific deployment scenarios, while the experimental conditions identifying when ML-DS helps versus harms inform design decisions.

Looking forward, this work implies that effective AI decision support requires substantial investment in stakeholder education and system transparency. Organizations implementing ML-DS should validate decision makers' understanding before deployment and continuously monitor whether support systems are actually improving outcomes. The framework provides tools for this validation, making it valuable for designing safer AI-human collaboration systems across industries.

• →ML predictions can influence decision makers' beliefs through Bayesian inference, with downstream effects on causal estimation and outcomes.
• →Even perfectly accurate ML models can harm decision-making when agents hold misaligned prior beliefs about how the system works.
• →Effective AI decision support requires accurate mental models from human decision makers, not just accurate ML models.
• →Researchers provide tractable solutions for analyzing Bayesian inference in linear Gaussian settings with ML-based decision support.
• →Organizations must validate stakeholder understanding and monitor actual outcome improvements before deploying ML-DS in high-stakes domains.