A Fixed-Budget, Cluster-Aware Standard for LLM-as-a-Judge Evaluation: A Multi-Hop RAG Stress Test

Researchers propose a standardized measurement protocol for evaluating retrieval-augmented generation (RAG) systems using LLM judges, addressing inconsistencies in how semantic search quality is assessed. The standard fixes key variables like evidence budget and prompt while requiring cluster-aware statistical testing, revealing that previous comparisons may have overstated progress and that traditional BM25 retrieval outperforms pure semantic methods under controlled conditions.

The paper tackles a fundamental problem in AI evaluation: the lack of rigor in how large language model judges compare RAG system performance. Current practice allows researchers to cherry-pick scoring criteria, evidence limits, and statistical tests, making it difficult to determine whether improvements reflect genuine algorithmic advances or measurement artifacts. This reproducibility crisis undermines the credibility of published benchmarks, particularly in multi-hop question answering where evidence assembly becomes complex.

The proposed standard addresses this by fixing critical variables that previously varied across studies: the retrieval candidate pool size, token budgets for context, answer length caps, the generator model, and prompt formulations. Critically, it mandates cluster-aware statistical inference rather than naive hypothesis tests that ignore data dependencies in benchmark datasets. The stress test using GADMEC reveals how dramatically these methodological choices alter conclusions—a binomial test initially suggests four significant improvements, but cluster-corrected analysis reduces this to one, demonstrating severe false-positive rates in current practices.

The finding that BM25 lexical retrieval outperforms pure semantic methods challenges the field's recent push toward embedding-based search. This suggests that without proper baselines and controlled conditions, semantic methods may receive undue credit. A hybrid lexical-semantic approach performs better, indicating complementary strengths. For practitioners building RAG systems, this implies that simple keyword matching remains highly competitive when evaluated fairly, potentially saving computational costs.

The paper's broader impact lies in setting precedent for evaluation rigor. Adoption of this standard would reduce false claims of progress and require researchers to pre-register hypotheses, improving scientific integrity. Organizations publishing RAG benchmarks should expect increasing pressure to conform to cluster-aware protocols.

• →Current LLM-as-judge evaluation of RAG systems lacks standardization, allowing researchers to manipulate results through variable choices in evidence budgets, prompts, and statistical tests.
• →Cluster-aware statistical testing reveals that previous benchmark comparisons vastly overstated progress, with false-positive rates masking when improvements are not actually significant.
• →Traditional BM25 keyword retrieval outperforms pure semantic retrieval methods under fair, budget-controlled conditions, challenging the field's embedding-first approach.
• →A proposed minimum measurement standard fixes critical variables and requires pre-registered hypotheses and second-judge replication to ensure reproducible RAG evaluation.
• →Adoption of rigorous evaluation standards could reduce computational waste by demonstrating that simpler hybrid lexical-semantic approaches outperform costly pure semantic methods.