Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study

Researchers develop strategies for extending large language models as evaluation tools to multilingual settings, addressing challenges in low-resource languages. The study reveals that fine-tuned smaller models match proprietary performance when in-domain data exists, while larger zero-shot models excel in out-of-domain scenarios, providing practical guidance for building multilingual evaluation systems.

This research addresses a significant gap in AI infrastructure by tackling the challenge of automating text evaluation across languages. While LLM-based evaluation has become standard practice in English-language NLP research, extending these systems to multilingual contexts remains underdeveloped despite growing global demand. The study's systematic analysis across English, Spanish, and Basque—spanning high-, mid-, and low-resource languages—provides empirical evidence for building more inclusive evaluation pipelines.

The findings reveal important trade-offs that practitioners must navigate. The availability of in-domain training data fundamentally changes the optimal approach: organizations with sufficient labeled data can deploy smaller, fine-tuned models efficiently, reducing computational costs and infrastructure requirements. Conversely, organizations without such data should rely on larger pre-trained models in zero-shot settings. The cautionary finding that out-of-domain fine-tuning can degrade performance highlights the importance of data quality over quantity in this context.

For the broader AI development community, this research enables more equitable evaluation practices. Low-resource language communities have historically received less attention in AI research partly because evaluation infrastructure lags behind high-resource languages. By providing open-source code and extended meta-evaluation datasets, the researchers democratize access to reliable multilingual evaluation tools. This supports fairer assessment of language models across linguistic diversity.

Looking forward, organizations building multilingual AI systems should assess their available in-domain data before selecting evaluation approaches. The research suggests a practical framework: audit existing training resources, evaluate model size requirements, and test translation-based versus native approaches. As AI systems increasingly serve global markets, investment in robust multilingual evaluation infrastructure becomes a competitive necessity rather than an afterthought.

• →Fine-tuned smaller models achieve proprietary-level performance when in-domain evaluation data is available, reducing computational requirements
• →Zero-shot evaluation with larger models outperforms fine-tuned smaller models in out-of-domain settings across languages
• →Fine-tuning on out-of-domain data degrades rather than improves model performance, making data selection critical
• →Multilingual evaluation tools are now publicly available, enabling fairer AI assessment across English, Spanish, and low-resource languages
• →The research provides practical guidance for organizations building language-agnostic evaluation pipelines without extensive labeled data