Offline reward shaping with scaling human preference feedback for deep reinforcement learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reward function design is crucial for aligning AI behavior with human intent.
• Preference-based Reinforcement Learning (PbRL) uses human preferences for reward learning.
• Current PbRL methods often lack preference intensity and dynamic feedback.

Purpose of the Study:

• To develop novel feedback methods for more accurate human preference learning in PbRL.
• To improve the alignment of learned reward functions with human intent.
• To address limitations of non-dynamic and intensity-agnostic feedback in existing PbRL.

Main Methods:

• Proposed the scaling preference (SP) feedback method.
• Introduced the qualitative and quantitative scaling preference (Q2SP) feedback method.
• Implemented and evaluated methods using offline data on benchmark control and robotic tasks.

Main Results:

• SP and Q2SP methods enable humans to express the degree of preference between trajectories.
• More detailed feedback significantly improves the accuracy of learned reward functions.
• Experimental results show competitive performance against baseline and state-of-the-art approaches.

Conclusions:

• Novel feedback methods enhance reward learning accuracy in PbRL.
• Detailed preference feedback is key to better aligning AI with human intent.
• The proposed methods offer a promising advancement for human-AI collaboration in complex tasks.