AIBullisharXiv – CS AI · Jun 96/10
🧠
Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
A new study demonstrates that pairwise comparison methods like Elo, commonly used to evaluate generative AI models, produce rankings that correlate strongly (>0.9 Spearman correlation) with ground-truth accuracy benchmarks. The research shows these comparative evaluations substantially outperform direct judging when evaluators are weak and are largely resistant to stylistic bias and judge preference, though minor effects like answer repetition can influence outcomes.