Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

Researchers propose a Stackelberg game framework for optimizing reward models in large language model alignment, addressing the suboptimality of standard KL-regularized reward optimization. A simple reward shaping scheme improves inference-time alignment by reducing base policy bias while mitigating reward hacking risks, demonstrating 66%+ win rates against baselines.

This research tackles a fundamental problem in LLM safety: the tension between faithfully optimizing for user preferences and maintaining stability through KL regularization. Standard alignment approaches apply learned reward models directly while constraining deviation from base policies, but this conservative approach inadvertently preserves undesirable behaviors from the original model. The paper identifies this as a game-theoretic problem where reward model design becomes the strategic variable, not just a static function to optimize against.

The Stackelberg game formulation elegantly captures the hierarchical relationship between reward design and policy optimization. By framing reward shaping as a mechanism design problem, the authors move beyond treating the reward model as fixed and instead optimize it to better guide policy learning under realistic constraints. This theoretical insight has practical value because it suggests simple, implementable modifications to existing pipelines.

The empirical results matter significantly for practitioners deploying alignment techniques. Achieving consistent improvement with minimal computational overhead makes this approach immediately useful for production systems. The 66%+ win rate across diverse evaluation settings suggests the method generalizes robustly, addressing concerns about context-dependent alignment failures.

For the broader AI safety ecosystem, this work demonstrates how game theory provides tools for understanding alignment tradeoffs. As LLMs become more capable, the gap between what reward models specify and what users actually prefer will only widen. Methods that optimize the specification itself, rather than just optimizing against fixed specifications, represent a meaningful direction forward.

• →Reward shaping via Stackelberg game theory improves LLM alignment by reducing base policy bias while preventing reward hacking tradeoffs.
• →The method integrates seamlessly into existing alignment frameworks with minimal computational overhead.
• →Empirical validation shows 66%+ win rates compared to standard baselines across multiple evaluation settings.
• →Game-theoretic reward model design offers practical improvements over static reward optimization approaches.
• →This approach scales inference-time alignment improvements, directly benefiting deployed language model systems.