Adaptive querying for reward learning from human feedback

Area of Science:

• Robotics
• Artificial Intelligence
• Human-Robot Interaction

Background:

• Learning from human feedback is crucial for robot adaptation and safety.
• Current methods often rely on a single feedback format, limiting learning potential.
• Optimizing both query states and feedback modalities can improve robot learning efficiency.

Purpose of the Study:

• To develop and evaluate an adaptive feedback selection method for learning robot behaviors from human input.
• To investigate the benefits of using multiple feedback formats and optimizing query states.
• To enhance the sample efficiency and effectiveness of robot training through human feedback.

Main Methods:

• An iterative, two-phase approach: critical state selection followed by information-gain-based feedback format selection.
• Incorporation of feedback cost and probability into the feedback format selection process.
• Experimental validation in simulation and with a physical robot.

Main Results:

• Demonstrated sample efficiency in learning to avoid undesirable behaviors in simulation.
• User studies confirmed the practicality and effectiveness of the adaptive feedback selection method.
• The approach successfully accelerated robot learning by seeking informative, user-aligned feedback.

Conclusions:

• Adaptive feedback selection significantly improves robot learning from human input by leveraging multiple feedback formats.
• Optimizing query states and feedback modalities is key to efficient and effective human-robot learning.
• This method offers a practical and effective solution for enhancing robot safety and user alignment.