Off-Policy Evaluation with Strategic Agents via Local Disclosure

Researchers propose a novel off-policy evaluation method that addresses strategic behavior by agents who modify their characteristics in response to policies. By leveraging post-hoc explanations to reveal pre-strategic information, the approach mitigates covariate shifts and enables more accurate policy assessment in one-shot settings with incomplete knowledge of agent responses.

This research tackles a fundamental challenge in decision-making systems: evaluating policies when subjects strategically alter their observable characteristics. Traditional off-policy evaluation assumes covariates remain independent of policy choices, an assumption violated when agents anticipate and respond to decisions. The authors' contribution is particularly relevant for high-stakes domains like lending, hiring, and resource allocation where strategic behavior is common.

The key innovation centers on using local disclosure mechanisms—post-hoc explanations of decisions—to reveal agents' true characteristics before they adapted strategically. This transforms an information asymmetry problem into one with recoverable structure. By modeling agent response behavior through conditional log-normal distributions and constructing doubly robust estimators, the framework provides theoretical consistency guarantees while remaining practical for one-shot interactions with partial information.

For practitioners deploying automated decision systems, this work has meaningful implications. Current evaluation methods may systematically overestimate or underestimate policy performance when agents respond strategically. Better OPE translates to more reliable policy comparisons and fairer outcomes. The approach also demonstrates how strategic behavior, rather than purely obstructing evaluation, can be partially recovered through thoughtful system design.

The research bridges machine learning and mechanism design by showing that transparency mechanisms serve dual purposes: improving fairness perceptions and enabling better empirical evaluation. Future work may extend this framework to multi-round interactions or explore other disclosure designs that maximize information recovery while maintaining decision-maker objectives.

• →Off-policy evaluation becomes unreliable when agents strategically modify covariates in response to policies, violating standard exogeneity assumptions.
• →Post-hoc explanations can reveal pre-strategic agent characteristics, partially recovering information lost through strategic adaptation.
• →The proposed doubly robust estimator provides theoretical consistency guarantees under conditional log-normal response distributions.
• →Interaction design and transparency mechanisms can simultaneously improve fairness and enable more accurate policy evaluation.
• →This framework applies to one-shot settings with only partial knowledge of agent response behavior, increasing practical applicability.