Poster
in
Workshop: Bayesian Decision-making and Uncertainty: from probabilistic and spatiotemporal modeling to sequential experiment design
Exploring and Addressing Reward Confusion in Offline Preference Learning
Xin Chen, Cynthia · Sam Toyer · Florian Shkurti
Keywords: [ Preference Learning; Causal Confusion; Active Learning ]
Abstract:
Spurious correlations in a reward model's training data can prevent Reinforcement Learning from Human Feedback (RLHF) from identifying the desired goal and induce unwanted behaviors. In this work, we study the reward confusion problem in offline RLHF where spurious correlations exist in data. We create a lightweight benchmark to study this problem and propose a method that can reduce reward confusion by leveraging model uncertainty and the transitivity of preferences with active learning.
Chat is not available.