Completion vs Optimality: Policy Gradient in Long-Horizon Cumulative-Damage Problems

Researchers identify critical failure modes in policy-gradient reinforcement learning methods when applied to long-horizon problems with cumulative damage, where short-term attractive actions lead to long-term negative outcomes. The study proposes a decomposition framework separating completion (reaching terminal horizon) from optimality (achieving dynamic-programming benchmarks) and validates predictions across two distinct domains: career planning and sports performance.

This research addresses a fundamental challenge in reinforcement learning optimization: the tension between solving exploration-exploitation tradeoffs and avoiding locally greedy decisions that compound into globally suboptimal outcomes. Policy-gradient methods like PPO struggle with long-horizon cumulative-damage problems because reward signals become obscured by the complexity of multi-step consequences. The researchers' contribution lies in diagnosing two orthogonal failure modes and proposing architectural solutions through decomposition. By introducing horizon access constraints and action-space restrictions, they demonstrate measurable improvements in task completion rates, though optimality gaps persist due to early-phase greedy commitment patterns. The dual-domain validation—comparing a 49-step bricklayer career simulation against a 20-season NBA forward career model—strengthens the generalizability claims by showing consistent prediction replication across disparate problem structures. However, the framework reveals horizon-dependent boundary effects (the H* transition around steps 6-14) indicating scale-dependent failure modes. This work has implications for autonomous systems navigating delayed-consequence environments: medical treatment sequencing, infrastructure maintenance scheduling, and long-term resource management. The identification of first-phase commitment bias suggests that reinforcement learning agents may require explicit mechanisms to defer early-stage optimization decisions. Future applications in industrial reinforcement learning should incorporate these decomposed failure modes to design more robust training curricula and reward structures for multi-decade or multi-century planning horizons.

• →Policy-gradient methods exhibit two distinct failure modes in long-horizon cumulative-damage problems: completion failure and optimality gap, requiring separate architectural solutions.
• →Linear soft penalties under PPO drive dominant-activity share toward zero, reducing completion rates unless combined with action-space restrictions.
• →First-phase greedy commitment at problem origin causes persistent optimality gaps even when task completion is achieved.
• →Horizon-invariant predictions replicate qualitatively across bricklayer and NBA domains, validating the framework's generalizability.
• →Horizon-dependent boundary effects emerge around H*∈[6,14] under tested parameters, indicating scale-sensitive optimization dynamics.