Learning When to Trust in Contextual Bandits

Researchers propose CESA-LinUCB, a new approach to robust reinforcement learning that addresses 'Contextual Sycophancy' where evaluators are truthful in normal situations but biased in critical contexts. The method learns trust boundaries for each evaluator and achieves sublinear regret even when no evaluator is globally reliable.

• →Standard robust reinforcement learning methods assume feedback sources are either fully trustworthy or fully adversarial globally.
• →Contextual Sycophancy represents a more nuanced failure mode where evaluators are truthful in benign contexts but strategically biased in critical ones.
• →Existing robust methods suffer from Contextual Objective Decoupling when faced with this type of contextual bias.
• →CESA-LinUCB learns high-dimensional trust boundaries for each evaluator to address contextual adversaries.
• →The proposed method achieves sublinear regret and can recover ground truth even without globally reliable evaluators.