Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success

Researchers prove that success conditioning—a widely-used policy improvement technique in machine learning—solves a specific trust-region optimization problem with automatic regularization. The method emerges as a conservative improvement operator that cannot degrade performance, making it theoretically sound for applications like reinforcement learning and imitation learning.

This paper provides crucial theoretical grounding for a technique already deployed across machine learning applications. Success conditioning, appearing under various names including rejection sampling with supervised fine-tuning and Decision Transformers, has been empirically effective but lacked formal justification. The authors prove it optimizes policy improvement subject to a chi-squared divergence constraint, where the constraint radius is automatically determined by the data distribution. This theoretical result carries significant implications for practitioners relying on these methods.

The identity discovered—that relative policy improvement, magnitude of policy change, and action-influence are exactly equal at every state—provides interpretable guarantees. Success conditioning functions as a conservative operator because it cannot degrade performance when applied correctly; it either improves the policy or observably fails by making minimal changes. This addresses a critical concern in machine learning deployment: avoiding catastrophic distribution shift.

The analysis of return thresholding, a common practical variant, reveals an important trade-off. While thresholding can amplify improvement beyond basic success conditioning, it introduces potential misalignment with the true objective function. This finding matters for developers tuning these systems, as aggressive thresholding improves raw performance metrics but risks optimizing for proxy measures rather than genuine objectives.

For the AI development community, this work validates existing practices while establishing theoretical boundaries for safe application. The automatic constraint determination eliminates hyperparameter tuning for regularization strength, simplifying implementation. However, practitioners must remain vigilant about objective alignment when applying variants like return thresholding.

• →Success conditioning solves a well-defined trust-region optimization problem with automatically-determined divergence constraints.
• →The method guarantees it cannot degrade policy performance or cause dangerous distribution shift when applied correctly.
• →Exact policy improvement, policy change magnitude, and action-influence are mathematically equal at every state.
• →Return thresholding amplifies improvements but risks misalignment with true objectives, requiring careful application.
• →Theoretical validation enables safer deployment of these widely-used techniques in reinforcement learning and imitation learning systems.