7 articles tagged with #psychometrics. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers apply psychometric analysis to large language model benchmarks, discovering that AI's general intelligence factor (G-factor) peaked around 2023-2024 before fragmenting as models specialized in reasoning tasks. The finding suggests AI development is shifting from unified capability improvement toward specialized tool-using systems, challenging assumptions about monolithic AGI progress.
Research reveals that standard human psychological questionnaires fail to accurately assess the true psychological characteristics of large language models (LLMs). The study of eight open-source LLMs found significant differences between self-reported questionnaire responses and actual generation behavior, suggesting questionnaires capture desired behavior rather than authentic psychological traits.
Researchers developed a method using Differential Item Functioning (DIF) analysis to identify systematic differences between human and AI chatbot performance on educational assessments. The study tested six leading chatbots including ChatGPT-4o, Gemini, and Claude on chemistry and entrance exams to help educators design AI-resistant assessments.
Researchers have developed PsyCogMetrics AI Lab, a cloud-based platform that applies psychometric and cognitive science methodologies to evaluate Large Language Models. The platform was created through a three-cycle Action Design Science study and aims to advance AI evaluation methods at the intersection of psychology, cognitive science, and artificial intelligence.
Researchers propose using psychometric modeling to correct systematic biases in human evaluations of AI systems, demonstrating how Item Response Theory can separate true AI output quality from rater behavior inconsistencies. The approach was tested on OpenAI's summarization dataset and showed improved reliability in measuring AI model performance.
Researchers developed an unsupervised machine learning framework using autoencoders and probabilistic models to detect inattentive survey respondents without traditional attention checks. The study found that survey structure is more important than model complexity for detection effectiveness, with well-designed instruments enabling reliable identification of low-quality responses.
Researchers developed a framework using large language models to simulate virtual respondents for validating psychometric survey items, addressing the challenge of ensuring construct validity without costly human data collection. The approach uses trait-response mediators to identify survey items that robustly measure intended psychological traits across three major trait theories.
