The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence

Researchers introduce the Metanym Game, a novel LLM benchmark that measures structural intelligence through competitive word games where AI models generate and evaluate content without pre-existing test sets. Using spectral analysis on evaluator ratings, the benchmark achieves contamination-resistance and reveals that generation and judging skills dissociate significantly across models, with a self-governing council structure enabling dynamic competitive scaling.

The Metanym Game represents a methodological breakthrough in LLM evaluation, addressing a critical weakness in current benchmarking practices: test contamination through training data leakage. By having models generate all test content rather than answering predetermined questions, researchers eliminate this vulnerability entirely while introducing a peer-review mechanism that extracts competence metrics from rating patterns alone.

Traditional LLM benchmarks rely on fixed test sets and human-annotated ground truth, creating dual problems. Models can memorize answers during training, and establishing factual accuracy requires expert "oracle" judges whose availability limits scalability. The Metanym Game sidesteps both issues through an elegant mathematical solution: singular value decomposition of the ratings matrix simultaneously quantifies each model's competence as both a content creator and evaluator, achieving Pearson r=0.92 correlation with GPQA Diamond without needing predetermined correct answers.

The finding that generation and judgment skills dissociate—strongest generators prove mediocre judges—has profound implications for AI safety and alignment research. It suggests capability and wisdom don't correlate, meaning raw intelligence doesn't guarantee sound decision-making in governance contexts. This directly impacts how organizations should structure AI oversight and which models should influence deployment decisions.

The contestable council structure creates a dynamic, self-correcting benchmark that remains resistant to gaming over time. As models improve, stronger performers earn seats automatically, preventing institutional decay. For the AI research community, this work establishes a template for building evaluation systems that scale without compromising integrity, crucial as models become increasingly capable at potentially exploiting evaluation frameworks.

• →The Metanym Game eliminates training data contamination by having models generate all test content rather than answering fixed questions.
• →Singular value decomposition of ratings matrices enables factual accuracy assessment without human oracles or ground truth labels.
• →Generation and judgment skills dissociate significantly across LLMs, suggesting capability doesn't guarantee reliable decision-making.
• →Self-governed contestable council structure enables benchmark scaling while remaining resistant to gaming and institutional degradation.
• →Achieves 0.92 Pearson correlation with GPQA Diamond while being entirely self-contained and temporally stable.