Towards a Sharp Analysis of Offline Policy Learning for $f$-Divergence-Regularized Contextual Bandits

Researchers achieved breakthrough sample complexity improvements for offline reinforcement learning algorithms using f-divergence regularization, particularly for contextual bandits. The study demonstrates optimal O(ε⁻¹) sample complexity under single-policy concentrability conditions, significantly improving upon existing bounds.

• →First achievement of O(ε⁻¹) sample complexity under single-policy concentrability for reverse KL divergence regularized contextual bandits.
• →Novel pessimism-based analysis surpasses existing bounds, improving from O(ε⁻²) to O(ε⁻¹) under single-policy conditions.
• →For strongly convex f-divergences, sharp O(ε⁻¹) complexity is achievable without pessimistic estimation or single-policy concentrability.
• →Research provides near-matching lower bounds demonstrating necessity of multiplicative dependency on single-policy concentrability.
• →Findings extend to contextual dueling bandits and advance theoretical understanding of f-divergence regularization objectives.