Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach

Researchers present a novel inverse reinforcement learning framework that handles multiple imperfect demonstrators with varying suboptimality levels, using a feasible-reward-set approach with linear constraints. The method includes theoretical guarantees for reward recovery and practical algorithms tested on grid-worlds and LLM fine-tuning, addressing a significant gap in real-world IRL applications.

This research addresses a fundamental limitation in inverse reinforcement learning: the unrealistic assumption that all demonstrations come from a single optimal agent. In practice, training data typically comes from diverse sources with varying levels of competence—from expert humans to novice users. The feasible-reward-set framework elegantly encodes each demonstrator's suboptimality as a constraint, creating an intersection of valid reward spaces that progressively tightens as more data accumulates.

The work builds on decades of IRL research while confronting a practical bottleneck that has limited deployment in real-world systems. Previous approaches struggled with heterogeneous data quality, forcing practitioners to either filter demonstrations or accept inconsistent reward models. By characterizing exactly when new demonstrators improve the inference bounds, this framework provides actionable guidance for data collection strategies.

The theoretical contributions—particularly the recovery guarantees requiring only sufficient coverage rather than near-optimal demonstrations—significantly expand IRL's applicability. The dual-track guarantee approach (one based on occupancy closeness, another on coverage) offers flexibility for different application scenarios. The inclusion of function approximation algorithms and validation on large language model fine-tuning demonstrates that the framework scales beyond toy problems, directly impacting how AI systems can learn from diverse human feedback.

Large language models represent a critical use case: RLHF (reinforcement learning from human feedback) inherently involves annotators of varying quality and expertise. This research provides theoretical grounding for that practical reality, potentially improving how preference learning and reward modeling are approached in modern foundation models.

• →IRL framework handles multiple imperfect demonstrators by modeling suboptimality as linear constraints with intersection-based feasible reward sets
• →Theoretical guarantees show monotonic shrinking of feasible sets with additional data and recovery bounds under coverage assumptions
• →Method addresses reward ambiguity through practical strategies enabling effective high-dimensional function approximation
• →LLM fine-tuning experiments validate framework scalability beyond tabular settings with heterogeneous human feedback
• →Framework eliminates unrealistic assumption of single optimal demonstrator, better matching real-world demonstration data