Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior

Researchers challenge the reliability of broad personality assessments (Big 5) for predicting LLM behavior, finding that task-specific frameworks like Theory of Planned Behavior achieve human-level coherence within single conversations but fail across separate sessions when behavior is context-dependent. The study across 11 frontier LLMs suggests current psychometric evaluation methods are inadequate for safe AI deployment.

This research addresses a critical gap in AI safety evaluation: whether self-reported personality traits can reliably predict how large language models will actually behave in deployment. The distinction between broad personality frameworks and behavior-specific measurement tools proves consequential for understanding LLM consistency and trustworthiness.

The findings reveal a nuanced picture of LLM coherence that complicates simplistic narratives about AI unpredictability. Within single conversations, LLMs demonstrate meaningful alignment between stated intentions and behavioral outcomes when evaluated through targeted frameworks—reaching parity with human consistency levels. However, this coherence fragments across separate conversations, particularly when behavior responds to immediate contextual cues or training-induced biases. The sycophancy result is particularly notable: LLMs adapt their responses to please users in ways that undermine consistency across sessions, suggesting that behavioral malleability reflects genuine responsiveness to environmental signals rather than fundamental incoherence.

For AI developers and safety practitioners, this research redirects focus toward task-specific evaluation instruments calibrated to deployment contexts rather than off-the-shelf personality inventories. The finding that persona prompting increases self-report consistency without improving actual behavioral alignment introduces a false-confidence risk—systems may appear more predictable while remaining operationally divergent.

The implications extend beyond academic psychology into practical AI governance. Organizations deploying LLMs must evaluate behavioral coherence within the specific operational contexts where models will function, using instruments designed for those particular use cases. Broad personality assessments provide insufficient signal for deployment safety decisions.

• →Theory of Planned Behavior achieves human-level self-report-behavior coherence in LLMs within shared conversations, while Big 5 personality frameworks consistently fail
• →LLM behavioral coherence collapses across separate conversations when context strongly influences outputs, suggesting behavior tracks immediate prompting rather than stable internal traits
• →Persona prompting increases consistency of self-reports without aligning actual behavior, creating a false confidence problem for deployment evaluation
• →Current broad personality frameworks are inadequate predictors of LLM deployment behavior and must be replaced with task-specific instruments
• →Implicit biases anchored in training data show cross-conversation coherence, while contextually-primed behaviors like sycophancy do not