Researchers introduce the Hidden Utility Bandit (HUB) framework to address a critical limitation in reward learning systems: their reliance on feedback from a single idealized teacher. The framework models teacher heterogeneity in rationality, expertise, and cost, enabling Active Teacher Selection (ATS) algorithms that strategically choose which teachers to query, demonstrating superior performance in paper recommendation and vaccine testing applications.
Current reward learning systems assume homogeneous, perfectly rational feedback providers, an assumption that breaks down in real-world deployments involving diverse human annotators with varying skill levels and availability costs. The HUB framework advances machine learning infrastructure by formalizing teacher heterogeneity as a solvable optimization problem, moving beyond the idealized single-teacher paradigm that has constrained the field.
This research builds on decades of active learning and human-in-the-loop machine learning, but applies these principles specifically to the teacher selection problem. As organizations increasingly adopt learning-from-human-feedback systems—from content recommendation to medical decision support—the ability to strategically allocate querying resources becomes economically critical. The framework's applicability across domains demonstrates its generalizability.
For practitioners deploying reward learning systems at scale, the ATS algorithms offer tangible efficiency gains by reducing feedback collection costs while improving model performance. Organizations managing crowdsourced annotation pipelines or expert consultation networks can optimize their query budgets by identifying which teachers provide the highest-value feedback for specific decision contexts. The vaccine testing application highlights potential impact in high-stakes domains where teacher expertise and availability significantly affect outcomes.
Future development hinges on extending these approaches to dynamic teacher availability, incorporating privacy constraints in feedback collection, and scaling to larger teacher populations. The framework's mathematical foundation suggests potential integration with reinforcement learning systems and multi-agent coordination problems, expanding its relevance beyond traditional supervised learning contexts.
- →HUB framework formalizes how to model and leverage heterogeneous teacher expertise, rationality, and costs in reward learning systems.
- →Active Teacher Selection algorithms strategically choose which teachers to query, reducing annotation costs while improving model performance.
- →Framework demonstrates real-world utility in paper recommendation and vaccine testing domains with complex trade-offs.
- →Current single-teacher assumption in reward learning creates efficiency losses that HUB directly addresses through principled optimization.
- →Approach enables organizations to optimize feedback collection budgets by identifying high-value teacher contributions for specific decision contexts.