When LLM Judge Scores Look Good but Best-of-N Decisions Fail

Research reveals that large language models used as judges for scoring responses show misleading performance when evaluated by global correlation metrics versus actual best-of-n selection tasks. A study using 5,000 prompts found that judges with moderate global correlation (r=0.47) only captured 21% of potential improvement, primarily due to poor within-prompt ranking despite decent overall agreement.

• →LLM judges with moderate global correlation (r=0.47) only achieve 21% of optimal performance in best-of-n selection tasks.
• →Global agreement metrics are misleading because they're driven by prompt-level baseline effects rather than within-prompt ranking ability.
• →Within-prompt correlation is significantly lower (r=0.27) than global correlation, indicating poor relative ranking performance.
• →Coarse pointwise scoring creates ties in 67% of pairwise comparisons, reducing selection effectiveness.
• →Explicit pairwise judging improves performance recovery from 21.1% to 61.2% in best-of-2 scenarios.