Solving robotics tasks with prior demonstration via exploration-efficient deep reinforcement learning

Area of Science:

• Robotics
• Artificial Intelligence
• Machine Learning

Background:

• Deep reinforcement learning (RL) in robotics often suffers from inefficient exploration and bootstrapping errors.
• Imitation bootstrapped reinforcement learning (IBRL) offers a foundation but can be further optimized.

Purpose of the Study:

• To propose an exploration-efficient deep reinforcement learning with reference (DRLR) policy framework.
• To enhance the IBRL algorithm by modifying the action selection module to mitigate bootstrapping errors.
• To improve policy convergence and prevent overfitting in complex robotics tasks.

Main Methods:

• Developed a DRLR framework based on IBRL, modifying the action selection module for calibrated Q-values.
• Integrated Soft Actor-Critic (SAC) as the RL policy to prevent convergence to sub-optimal policies, replacing Twin Delayed Deep Deterministic Policy Gradient (TD3).
• Empirically validated the framework on bucket loading and open drawer robotics tasks in simulation.

Main Results:

• The modified action selection module successfully mitigated bootstrapping errors, leading to more efficient exploration.
• The use of SAC prevented policy overfitting and convergence to sub-optimal solutions.
• The DRLR framework demonstrated robustness across tasks with varying state-action dimensions and demonstration qualities.

Conclusions:

• The proposed DRLR framework significantly improves exploration efficiency and learning stability in robotics tasks.
• The framework's effectiveness was validated through simulation and successful deployment on a real-world industrial robotics task (bucket loading on a wheel loader).
• The sim-to-real transfer results confirm the practical applicability and robustness of the DRLR approach.