Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity

Researchers propose a new method for training large language models (LLMs) that addresses the diversity loss problem in reinforcement learning approaches. Their technique uses the α-divergence family to better balance precision and diversity in reasoning tasks, achieving state-of-the-art performance on theorem-proving benchmarks.

• →Current reinforcement learning methods for training LLMs cause significant loss in response diversity by concentrating on high-probability regions.
• →The proposed method uses explicit target distribution filtering to preserve relative probabilities of correct answers.
• →The α-divergence family approach enables direct control of the precision-diversity trade-off in model training.
• →The method achieved state-of-the-art performance on Lean theorem-proving benchmarks, particularly excelling in coverage metrics.
• →This research addresses a fundamental limitation in current LLM training methodologies for reasoning tasks.