BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

Researchers introduce BabelJudge, an open-source framework that audits LLM-as-a-judge systems for systematic biases including position bias, verbosity bias, and cross-lingual degradation. The benchmark reveals significant reliability gaps across languages, with performance dropping from 0.714 in Hindi to 0.550 in Swahili, and extends evaluation to agentic AI systems through trajectory-level perturbations.

BabelJudge addresses a critical blind spot in AI evaluation methodology. As LLM-as-a-judge has become the standard for scalable NLP evaluation, the field has largely ignored systematic biases that inflate apparent accuracy metrics. The framework's innovation lies in its gold-labeling approach through controlled perturbation, which eliminates expensive human annotation while exposing failure modes invisible to raw accuracy scores.

The cross-lingual findings are particularly concerning for AI deployment in lower-resource languages. A 0.164-point drop in bias-penalized reliability between Hindi and Swahili masks deeper issues: Swahili order consistency collapsed to 0.480, indicating near-random verdicts when response positions swap. This reveals that judges are not learning semantic quality but rather exploiting superficial patterns. The research contextualizes a broader problem in AI evaluation infrastructure—metrics designed for English-first development inherently disadvantage non-English applications.

For practitioners building LLM evaluation pipelines, BabelJudge provides actionable diagnostics rather than theoretical critiques. The framework's extension to agentic systems through trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls) acknowledges that evaluation methodologies must evolve alongside AI capabilities. Three new metrics—tool accuracy, hallucination detection rate, and trajectory-length bias—create more granular visibility into agent behavior.

The release as an open-source Python package supporting 11 judge backends democratizes access to reliability auditing. Organizations can now benchmark their evaluation systems before deploying them at scale, reducing the risk of systematic evaluation errors compounding through production pipelines.

• →BabelJudge reveals LLM judges suffer from position bias, verbosity bias, and order inconsistency invisible to standard accuracy metrics.
• →Cross-lingual evaluation shows 16.4-point reliability gaps between high-resource and low-resource languages, with Swahili order consistency near random at 0.480.
• →Gold-labeling through controlled perturbation eliminates annotation costs while enabling comprehensive bias auditing without human preference labels.
• →Framework extends to agentic AI evaluation via nine trajectory-level perturbations and three new metrics for tool use and hallucination detection.
• →Open-source release supports 11 judge backends, enabling organizations to audit evaluation system reliability before production deployment.