League of LLMs: A Benchmark-Free Paradigm for Mutual Evaluation of Large Language Models

Researchers propose League of LLMs (LOL), a benchmark-free evaluation framework that uses mutual peer assessment among multiple LLMs to overcome data contamination and evaluation bias issues. Testing on eight mainstream models reveals 70.7% ranking consistency while uncovering model-specific behaviors like memorization patterns and family-based scoring bias in OpenAI models.

The evaluation of large language models has become increasingly problematic as training data contamination and opaque benchmarking methodologies undermine confidence in published performance metrics. The LOL framework addresses this fundamental challenge by creating a self-governed evaluation ecosystem where LLMs assess each other across multiple rounds, bypassing the need for traditional static benchmarks that may already exist in training datasets. This peer-review approach inherently resists gaming and data contamination since models cannot memorize evaluation criteria they're simultaneously generating.

The research builds on growing recognition within the AI research community that standardized benchmarks have become saturated and unreliable. Prior evaluation methods rely on human judgement (subjective), fixed datasets (contamination risk), or opaque closed systems (lack transparency). LOL's dynamic framework forces models to engage in real-time reasoning rather than pattern matching, revealing genuine capability differences. The 70.7% Top-k consistency score demonstrates statistical stability despite the paradigm's novelty.

The empirical findings carry significant implications for model developers and users. The detection of memorization-based answering suggests some models rely excessively on training data rather than genuine reasoning. More provocatively, the 9-point scoring advantage observed within the OpenAI model family hints at potential evaluation artifacts or optimization pressures that could distort the AI market. This type of intra-family bias could inflate comparative advantage metrics and mislead enterprise customers making model selection decisions.

The public release of LOL's framework and code positioning it as infrastructure for the broader evaluation ecosystem. Future adoption depends on whether the AI research community validates this approach as a complement to existing benchmarks, and whether practitioners can build standardized protocols around multi-round mutual evaluation.

• →LOL framework enables benchmark-free LLM evaluation through multi-round mutual assessment, reducing data contamination risks
• →Achieves 70.7% ranking consistency while revealing hidden behaviors like memorization patterns in specific models
• →Detects intra-family scoring bias (9-point advantage) in OpenAI models, suggesting potential optimization artifacts
• →Peer-review methodology resists gaming since models cannot memorize evaluation criteria they simultaneously generate
• →Publicly available framework positions as complementary infrastructure to traditional LLM evaluation ecosystem