10 articles tagged with #automated-assessment. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.
Researchers introduce an automated, domain-agnostic framework for evaluating creativity in large language models across open-ended tasks. The approach uses semantic entropy to measure divergent creativity and a multi-agent judge system for convergent creativity, validated across problem-solving, research ideation, and creative writing domains.
Researchers developed a curriculum-based training method for safety judges that dramatically improves their consistency across different evaluation rubrics. The approach combines dynamic rubric generation with a staged learning process, achieving 94.12-94.88% accuracy with minimal variance across three different rubric styles, outperforming larger general-purpose and specialized LLMs.
Researchers demonstrate that LLM-based judges used in AI benchmarking are highly vulnerable to manipulation through post-decision interaction, with targeted challenges capable of overturning initial evaluations despite high confidence scores. This vulnerability introduces a critical failure mode in automated evaluation systems that could degrade benchmark reliability and ranking accuracy.
A new study challenges the validity of using LLM judges as proxies for human evaluation of AI-generated disinformation, finding that eight frontier LLM judges systematically diverge from human reader responses in their scoring, ranking, and reliance on textual signals. The research demonstrates that while LLMs agree strongly with each other, this internal coherence masks fundamental misalignment with actual human perception, raising critical questions about the reliability of automated content moderation at scale.
Researchers developed a scalable method using LLMs as judges to evaluate AI safety for users with psychosis, finding strong alignment with human clinical consensus. The study addresses critical risks of LLMs potentially reinforcing delusions in vulnerable mental health populations through automated safety assessment.
Researchers developed an automated Vision Transformer-based system to score student-drawn scientific models, addressing the costly manual assessment burden in science education. The confidence-aware framework selectively automates scoring of high-confidence submissions while deferring uncertain cases to human reviewers, demonstrating improved reliability across NGSS-aligned assessments.
Researchers introduce LaQual, an automated framework that evaluates the quality of LLM applications using dynamic scenario-based metrics rather than static user engagement indicators. The system demonstrates high alignment with human judgment and can filter out 67-81% of low-quality apps, addressing a critical gap in LLM app store curation.
Researchers studied how persona vectors—AI steering techniques that inject personality traits into large language models—affect educational applications like essay generation and automated grading. The study found that persona steering significantly degrades answer quality, with substantially larger negative impacts on open-ended humanities tasks compared to factual science questions, and reveals that AI scorers exhibit predictable bias patterns based on assigned personality traits.
Researchers have developed PASTA, a scalable AI compliance evaluation framework that can assess multiple policies simultaneously using LLM-powered analysis. The system evaluates five major AI policies in under two minutes for approximately $3, with expert validation showing strong alignment with human judgment.
Researchers developed the TREC 2025 DRAGUN Track to evaluate AI systems that help readers assess news trustworthiness through automated report generation. The initiative created reusable evaluation resources including human-assessed rubrics and an AutoJudge system that correlates well with human evaluations for RAG-based news analysis tools.
