Questionnaire Responses Do not Capture the Safety of AI Agents

Researchers argue that current AI safety assessments using questionnaire-style prompts on language models are inadequate for evaluating real AI agents. The study suggests these methods lack construct validity because LLM responses to hypothetical scenarios don't accurately represent how AI agents would actually behave in real-world deployments.

• →Standard AI safety assessments focus on unaugmented LLMs rather than AI agents that can perform actual behaviors and pose greater risks.
• →Questionnaire-style prompts create divergences in inputs, actions, environmental interactions, and processing compared to real agent behavior.
• →Current assessment methods make strong assumptions about LLMs' ability to accurately report their counterfactual behavior.
• →The researchers identify similar structural issues in current AI alignment approaches.
• →The study calls for improved safety assessments that address these construct validity shortcomings.