Human Psychometric Questionnaires Mischaracterize LLM Psychology: Evidence from Generation Behavior

Research reveals that standard human psychological questionnaires fail to accurately assess the true psychological characteristics of large language models (LLMs). The study of eight open-source LLMs found significant differences between self-reported questionnaire responses and actual generation behavior, suggesting questionnaires capture desired behavior rather than authentic psychological traits.

• →Standard human psychometric questionnaires mischaracterize LLM psychology compared to real-world generation behavior.
• →LLM responses to established questionnaires reflect desired behavior rather than stable psychological constructs.
• →Generation probability-based profiling provides more reliable assessment of LLM psychological characteristics than traditional questionnaires.
• →Established questionnaires may exaggerate demographic biases present in LLMs.
• →The findings challenge previous claims about consistent psychological dispositions in AI models.