Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment

Researchers revisited Best-of-N (BoN) sampling for AI alignment and found it's actually optimal when evaluated using win-rate metrics rather than expected true reward. They propose a variant that eliminates reward-hacking vulnerabilities while maintaining optimal performance.

• →Best-of-N sampling is computationally and statistically optimal for achieving high win-rates in inference-time alignment under practical conditions.
• →Previous theoretical work suggesting BoN was suboptimal focused on expected true reward metrics that may not reflect practical use cases.
• →Win-rate evaluation, based on pairwise comparisons, better aligns with how reward models are trained and evaluated in practice.
• →The researchers propose a simple variant of BoN that eliminates reward-hacking while maintaining optimal statistical performance.
• →Prior approaches are provably suboptimal when considering win-rate objectives, emphasizing the importance of appropriate evaluation metrics.