TAG: Teacher-Advice Mechanism With Gaussian Process for Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) faces challenges with sample inefficiency and exploration, especially in environments with sparse or delayed rewards.
• Existing learning from demonstration (LfD) methods often require extensive datasets, limiting their practical application.

Purpose of the Study:

• To develop a sample-efficient teacher-advice mechanism (TAG) that utilizes a small number of expert demonstrations to enhance RL.
• To improve agent exploration strategies in complex environments with limited data.

Main Methods:

• A teacher model is constructed using Gaussian processes to provide advice actions and confidence values.
• A guided policy is formulated to direct agent exploration based on the teacher's advice and confidence.
• The TAG mechanism is integrated with standard RL algorithms, such as soft actor-critic (SAC).

Main Results:

• The TAG mechanism significantly improves the performance and sample efficiency of typical RL algorithms.
• TAG enables more intentional and precise agent exploration, particularly in sparse reward settings.
• TAG-SAC achieved state-of-the-art results on complex continuous control tasks with delayed rewards.

Conclusions:

• The TAG mechanism offers a sample-efficient approach to learning from demonstrations for RL.
• TAG effectively addresses exploration challenges in RL, leading to substantial performance gains.
• This method shows promise for advancing RL applications in real-world scenarios requiring efficient learning from limited expert data.