Expert Behavior Prior Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Behavior prior reinforcement learning (BPRL) aims to boost sample efficiency in online reinforcement learning (RL) using offline demonstrations.
• Existing BPRL methods often use static datasets, leading to limited data diversity and suboptimal trajectories, which hampers policy exploitation and stability.

Purpose of the Study:

• To address limitations of static offline datasets in BPRL.
• To propose a novel algorithm, expert behavior prior (EBP), for improved online RL training.

Main Methods:

• Introduced a Q-guided conditional variational autoencoder (Q-CVAE) to generate expert policy priors from online replay buffers.
• Implemented an expert policy guidance (EPG) mechanism for selecting high-value actions.
• Integrated a policy gradient correction (PGC) module for stable policy improvement.

Main Results:

• The EBP algorithm demonstrated superior performance compared to state-of-the-art online RL methods.
• Achieved significantly higher sample efficiency and more stable convergence in experiments.
• Validated effectiveness across robotic control (Gym, PyBullet) and industrial control (DMControl) benchmarks.

Conclusions:

• EBP offers a more effective approach to policy priors by leveraging online data.
• The proposed methods enhance exploration, stability, and overall performance in online RL.
• EBP represents a significant advancement for practical applications of reinforcement learning.