Linear and Neural Dueling Bandits with Delayed Feedback

Researchers propose novel algorithms (LDB-DF and NDB-DF) for contextual dueling bandits that handle delayed feedback—a critical real-world constraint in recommender systems and LLM alignment. The breakthrough involves an Inverse Probability Weighting mechanism that eliminates bias from delayed observations, achieving theoretical regret bounds of O(d√T) for linear settings.

This research addresses a fundamental gap between theoretical bandit algorithms and practical deployment constraints. Standard contextual dueling bandit approaches assume immediate user feedback, an assumption violated consistently in production systems where preference judgments arrive asynchronously. The challenge extends beyond simple delay accommodation: unlike linear bandits with closed-form estimators, dueling bandit preference comparisons lack analytical solutions, making naive weighting adaptations mathematically biased. The proposed solution integrates Inverse Probability Weighting directly into the loss function, providing unbiased correction for stochastic delays without requiring closed-form estimators. This methodological contribution has broad implications for preference-based systems. Recommender platforms increasingly rely on dueling bandit frameworks for ranking optimization, where user feedback naturally arrives delayed and unevenly distributed. LLM alignment represents an emerging application domain where preference data collection involves costly human annotation inherently asynchronous with model training. The theoretical guarantees establish sub-linear regret bounds across both linear and neural settings, suggesting practical convergence properties. Real-world experiments validate effectiveness on both synthetic and production datasets, indicating readiness for implementation. The research materializes in a landscape where preference learning drives increasingly sophisticated AI systems. As language model deployment accelerates, efficient alignment mechanisms become critical infrastructure. The ability to handle delayed feedback without performance degradation directly translates to faster iteration cycles and reduced computational waste in model optimization.

• →Novel algorithms eliminate bias in dueling bandit feedback by integrating Inverse Probability Weighting into loss functions.
• →Theoretical analysis proves O(d√T) regret bounds for linear settings and sub-linear guarantees for neural dueling bandits with delays.
• →Solution addresses critical gap between idealized bandit algorithms and real-world deployment constraints in recommender systems.
• →Methods enable asynchronous preference learning, accelerating LLM alignment and recommendation optimization pipelines.
• →Empirical validation on simulated and real datasets demonstrates practical viability for production recommendation systems.