Human-in-the-Loop Reinforcement Learning in Continuous-Action Space.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Reinforcement learning (RL) often suffers from sample inefficiency.
• Existing human-in-the-loop RL (HRL) methods primarily address discrete action spaces.
• Continuous action spaces present unique challenges for efficient learning and human guidance.

Purpose of the Study:

• To propose a novel Q value-dependent policy (QDP)-based HRL algorithm (QDP-HRL) for continuous action spaces.
• To address sample inefficiency in RL by incorporating selective human expert advice.
• To enhance the learning speed and performance of agents in continuous control tasks.

Main Methods:

• Developed a QDP-HRL algorithm adapted to the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm.
• Implemented selective human advice based on the difference between twin Q-networks' outputs.
• Introduced an advantage loss function using expert experience and agent policy to guide critic network updates.

Main Results:

• QDP-HRL demonstrated significant improvements in learning speed across various continuous action space tasks.
• The algorithm achieved enhanced overall performance compared to baseline methods.
• Selective human intervention effectively reduced cognitive load while maximizing learning benefits.

Conclusions:

• QDP-HRL is an effective approach for improving sample efficiency in continuous action space RL.
• The proposed method offers a practical way to integrate human expertise into complex learning environments.
• This work advances HRL by extending its applicability to continuous control problems.