Evaluation of Large Language Models via Coupled Token Generation

Researchers propose a new method called coupled autoregressive generation to evaluate large language models more efficiently by controlling for randomness in their responses. The study shows this approach can reduce evaluation samples by up to 75% while revealing that current model rankings may be confounded by inherent randomness in generation processes.

• →Coupled autoregressive generation controls for randomness in LLM evaluation, requiring up to 75% fewer samples than traditional methods.
• →Current model rankings based on pairwise comparisons may be misleading due to confounding effects of generation randomness.
• →The research demonstrates that different evaluation approaches can lead to different model rankings even with infinite samples.
• →Experiments across Llama, Mistral, and Qwen model families validate the theoretical findings on benchmark efficiency.
• →The study suggests existing LLM evaluation protocols may not accurately reflect genuine model performance advantages.