Weak Human Preference Supervision for Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Robotics
• Machine Learning

Background:

• Reinforcement learning (RL) often requires a predefined reward function, which is difficult to specify for complex tasks.
• Current methods using human preferences for reward learning demand extensive, iterative human input, limiting scalability.
• Dynamic and efficient human feedback mechanisms are crucial for advancing RL applications.

Purpose of the Study:

• To develop a weak human preference supervision framework to reduce the cost and increase the efficiency of reward learning in RL.
• To introduce a human preference scaling model that captures nuanced human judgments of trajectory preferences.
• To establish a human-demonstration estimator for predicting preferences and minimizing direct human intervention.

Main Methods:

• Developed a human preference scaling model to quantify the degree of weak preferences between trajectory segments.
• Implemented a supervised learning approach for a human-demonstration estimator to predict preferences.
• Evaluated the framework on simulated robot locomotion tasks (MuJoCo games) comparing performance against fixed preference methods.

Main Results:

• The weak human preference supervision framework effectively solves complex RL tasks, achieving higher cumulative rewards than fixed preference methods.
• The human-demonstration estimator required human feedback for less than 0.01% of agent interactions.
• Human input costs were reduced by up to 30% compared to existing approaches.

Conclusions:

• The proposed framework offers a more natural and efficient approach to reward learning through weakly supervised learning.
• This method significantly reduces the need for extensive human input, making RL more practical for complex real-world problems.
• The approach shows promise for advanced RL systems, including human-autonomy teaming.