AINeutralarXiv – CS AI · 15h ago6/10
🧠
JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
Researchers introduce JuICE, a multilingual benchmark dataset revealing that current LLM-judges struggle to identify cultural errors in AI-generated responses, achieving only 52% F1 scores. The study demonstrates that LLMs fail to capture nuanced cultural contexts across diverse regions, suggesting existing evaluation methods inadequately assess cultural appropriateness in global AI deployment.