AdaGamma: State-Dependent Discounting for Temporal Adaptation in Reinforcement Learning

AdaGamma introduces a state-dependent discount factor method for deep reinforcement learning that learns to adjust discounting dynamically across different states, addressing instability issues in prior approaches through a return-consistency regularization objective. The method demonstrates empirical improvements when integrated into popular algorithms like SAC and PPO, with validated gains from real-world logistics deployment.

AdaGamma addresses a fundamental limitation in reinforcement learning where the discount factor—which balances planning horizon length against bootstrapping strength—remains fixed across all states despite conceptual arguments for adaptive adjustment. Traditional approaches to state-dependent discounting destabilize deep actor-critic architectures through TD-error collapse, where the learning process degenerates as target values become manipulated. The researchers solve this by pairing a learnable state-dependent discount function with a return-consistency objective that constrains the backup structure, preventing pathological learning dynamics.

The work progresses reinforcement learning toward more nuanced value estimation by recognizing that optimal discounting varies contextually. Early states might require longer planning horizons while terminal states need minimal bootstrapping. This insight has practical relevance since most production RL systems currently ignore this variation entirely. The theoretical contributions establishing well-posedness properties of the induced Bellman operator provide mathematical grounding often absent from pure empirical papers.

The empirical validation spans multiple dimensions: integration into both on-policy (PPO) and off-policy (SAC) algorithms demonstrates generality, continuous-control benchmarks show consistent improvements, and critically, a live A/B test on JD Logistics' platform proves real-world viability beyond toy problems. This production deployment matters significantly—academic RL papers rarely achieve validated commercial implementation, making this validation particularly noteworthy. The return-consistency regularization appears to be the key innovation preventing collapse, suggesting that preventing degenerate target manipulation through explicit constraints represents a tractable path forward for state-dependent adaptation in deep RL.

• →AdaGamma solves instability problems in state-dependent discount factors through return-consistency regularization, enabling practical deep reinforcement learning implementations.
• →The method successfully integrates into both on-policy (PPO) and off-policy (SAC) algorithms, showing broad applicability across RL paradigms.
• →Real-world deployment on JD Logistics platform validates statistically significant performance gains beyond academic benchmarks.
• →Theoretical analysis establishes well-posedness properties of the Bellman operator under state-dependent discounting, providing mathematical foundations for the approach.
• →Dynamic discount adjustment based on state context offers efficiency improvements by optimizing planning horizons contextually rather than using uniform values.