HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime

Researchers propose Hysteretic Policy Optimization (HPO), a refinement to GRPO reinforcement learning that addresses training instability in sparse-reward environments by downweighting negative-advantage updates and normalizing by mean length rather than per-response length. The adaptive variant (A-HPO) achieves 15% reward improvement over GRPO on benchmark tasks.

HPO addresses a fundamental challenge in reinforcement learning optimization where early training phases suffer from imbalanced advantage distributions. In sparse-reward settings, initial model outputs typically contain more failed attempts than successful ones, creating asymmetric gradient signals that destabilize training. The paper identifies a compounding issue: response-length normalization amplifies this effect by tying update magnitude to output length, causing longer failed responses to dominate gradient calculations. This represents a non-trivial engineering insight applicable across RL systems.

The proposed solution employs hysteresis—reducing weight on negative-advantage updates while equalizing length normalization across the batch. This resembles asymmetric loss weighting used in other domains, but tailored for policy optimization's specific dynamics. The adaptive variant eliminates hyperparameter tuning by automatically setting hysteretic weights based on observed advantage-sign ratios, improving practical applicability.

For the AI infrastructure space, this work has meaningful implications for training more sample-efficient language models and reasoning agents. Better sparse-reward training translates directly to reduced computational costs and accelerated development cycles for frontier models. The 15% improvement over GRPO on TeleLogs and consistent gains across 1.5B-7B model scales suggest broad applicability rather than task-specific utility.

The research validates through rigorous ablation studies, distinguishing genuine contributions from architectural choices. Future adoption depends on whether the community integrates HPO into standard RL frameworks and whether gains persist across diverse downstream tasks beyond the tested benchmarks.

• →HPO reduces training instability in sparse-reward RL by downweighting negative-advantage updates and using mean-length rather than per-response normalization
• →Adaptive HPO variant achieves 15% improvement over GRPO on TeleLogs benchmark without requiring manual hyperparameter tuning
• →Method shows largest gains during early training phases when sparse rewards create imbalanced advantage distributions
• →Technique applies across 1.5B-7B model scales, suggesting broad utility for AI development pipelines
• →Ablation studies confirm gains stem from balanced contribution of positive and negative advantages rather than architectural changes