#evaluation-systems News & Analysis

3 articles tagged with #evaluation-systems. AI-curated summaries with sentiment analysis and key takeaways from 50+ sources.

3 articles

AINeutralarXiv – CS AI · Jun 26/10

🧠

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

Researchers identify and address Perceptual Judgment Bias in multimodal large language models used as automated evaluators, where these models favor plausible narratives over visually accurate answers when text and images conflict. The team develops a training framework using perceptually perturbed datasets and reward modeling that improves MLLM judges' visual grounding and evaluation consistency across benchmarks.

AINeutralarXiv – CS AI · May 286/10

🧠

Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Researchers develop strategies for extending large language models as evaluation tools to multilingual settings, addressing challenges in low-resource languages. The study reveals that fine-tuned smaller models match proprietary performance when in-domain data exists, while larger zero-shot models excel in out-of-domain scenarios, providing practical guidance for building multilingual evaluation systems.

AIBullisharXiv – CS AI · Mar 126/10

🧠

Designing Service Systems from Textual Evidence

Researchers developed PP-LUCB, an algorithm that efficiently identifies optimal service system configurations by combining biased AI evaluation with selective human audits. The method reduces human audit costs by 90% while maintaining accuracy in selecting the best performing systems from textual evidence like customer support transcripts.