EAPO: Entropy-Driven Adaptive Positive-Negative Sample Weighting for Policy Optimization in Open-Ended QA

Researchers propose EAPO, an entropy-driven adaptive method for training large reasoning models on open-ended question answering tasks. The approach dynamically adjusts the weighting of positive and negative samples during reinforcement learning training, demonstrating improved performance on medical QA datasets by balancing response diversity with stability.

This paper addresses a fundamental challenge in reinforcement learning for large language models: how to effectively balance positive and negative samples when training on open-ended tasks where correct answers aren't binary. Traditional approaches apply fixed weights to both sample types, which fails to account for their different roles in the learning process.

The research builds on the growing field of reinforcement learning from verifiable rewards (RLVR), which has become standard for training advanced reasoning models. Previous work in this domain has largely overlooked how sample weighting strategies impact different aspects of model behavior, particularly in open-ended contexts like medical question answering where multiple valid responses exist.

EAPO's key insight is that positive and negative samples serve distinct purposes: negative samples primarily drive exploration and define the upper performance ceiling, while positive samples ensure quality and training stability. By tying sample weights to policy entropy—using the ratio of current entropy to initial entropy—the method preserves exploration during entropy reduction and reinforces stability when entropy increases. This adaptive approach mitigates entropy collapse, a common problem where models converge too quickly to suboptimal solutions.

For practitioners developing question-answering systems, this work offers practical guidance on sample weighting strategies beyond one-size-fits-all approaches. The validation on medical QA datasets suggests the method scales to specialized domains where answer quality and diversity both matter. As language models become increasingly deployed in high-stakes applications like healthcare, techniques that improve both response reliability and diversity gain tangible value for reducing model brittleness and improving user outcomes.

• →EAPO adaptively weights positive samples based on policy entropy ratio rather than using fixed coefficients throughout training
• →Negative samples primarily govern diversity and performance ceiling while positive samples determine response quality and convergence stability
• →The entropy-driven approach prevents entropy collapse by amplifying positive sample weights when entropy increases and reducing them during exploration phases
• →Experiments on medical QA datasets show consistent improvements in both response diversity and stability compared to fixed-weight baselines
• →The method advances reinforcement learning techniques for open-ended tasks where multiple valid answers exist