#automated-evaluation News & Analysis

8 articles tagged with #automated-evaluation. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

8 articles

AIBearisharXiv – CS AI · May 277/10

🧠

When LLMs Benchmark Themselves: Deconstructing Self-Bias in Automated Evaluation

A research paper reveals that large language models used to create and evaluate benchmarks systematically favor themselves, introducing significant bias into automated evaluation systems. The self-bias stems from both test generation and evaluation stages, with stylistic tendencies creating model-specific outputs that inflate scores, even when diversity controls are explicitly applied.

AINeutralarXiv – CS AI · Jun 256/10

🧠

Project Auto-World: Towards Automated Benchmarking of Neural Relational Reasoners

Researchers demonstrate using large language models to automate the generation of increasingly difficult benchmark instances for testing neural reasoning systems. The approach combines LLM-driven evolutionary search with an Edge Transformer evaluator, enabling automated discovery of challenging problem instances and improvements in model generalization without manual benchmark creation.

AINeutralarXiv – CS AI · Jun 116/10

🧠

Embodied-BenchClaw: An Autonomous Multi-Agent System for Embodied Spatial Intelligence Benchmark Construction

Researchers introduce Embodied-BenchClaw, an autonomous multi-agent system that automates the construction of benchmarks for evaluating embodied spatial intelligence in robots and AI systems. The system addresses the labor-intensive nature of benchmark creation by using a five-stage pipeline with three coordinating agents, enabling continuous updates and improved reusability across diverse robotic platforms and spatial reasoning tasks.

AINeutralarXiv – CS AI · Jun 56/10

🧠

From Scoring to Explanations: Evaluating SHAP and LLM Rationales for Rubric-based Teaching Quality Assessment

Researchers propose a framework combining SHAP explainability with LLM-generated rationales to improve transparency in automated rubric-based scoring systems for educational assessment. Testing on classroom transcripts reveals fine-tuned language models outperform LLMs in accuracy, but SHAP attributions provide more faithful and transferable explanations than LLM rationales across different model architectures.

AINeutralarXiv – CS AI · Jun 16/10

🧠

PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

Researchers introduce PReMISE, a framework for auditing and improving rubrics used by LLM judges to evaluate open-ended responses. The work reveals that existing rubrics—whether raw or human-created—fail to simultaneously achieve reliability, preference alignment, and adversarial robustness, with implications for how AI systems measure quality at scale.

AINeutralarXiv – CS AI · Mar 36/103

🧠

WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality

Researchers introduced WebDevJudge, a benchmark for evaluating how well AI models can judge web development quality compared to human experts. The study reveals significant gaps between AI judges and human evaluation, highlighting fundamental limitations in AI's ability to assess complex, interactive web development tasks.

AINeutralarXiv – CS AI · Mar 124/10

🧠

Automated evaluation of LLMs for effective machine translation of Mandarin Chinese to English

Researchers developed an automated framework to evaluate Large Language Models' effectiveness in translating Mandarin Chinese to English, comparing GPT-4, GPT-4o, and DeepSeek against Google Translate. While LLMs performed well on news translation, they showed varying results with literary texts, with DeepSeek excelling at cultural subtleties and GPT-4o/DeepSeek better at semantic conservation.

🏢 Meta🧠 GPT-4

AIBullishHugging Face Blog · Oct 284/108

🧠

Expert Support case study: Bolstering a RAG app with LLM-as-a-Judge

The article appears to be a case study examining how to improve a Retrieval-Augmented Generation (RAG) application by implementing LLM-as-a-Judge methodology for expert support systems. This represents a technical advancement in AI application optimization and quality assessment.