A Hybrid MPC for Constrained Deep Reinforcement Learning applied for Planar Robotic Arm

Area of Science:

• Robotics
• Artificial Intelligence
• Control Systems

Background:

• Deep reinforcement learning (DRL) excels in high-dimensional control but suffers from inefficient sample usage during exploration.
• Model-based algorithms learn environment dynamics to improve sample efficiency but often have lower asymptotic performance than model-free methods.
• Constraint satisfaction remains a challenge for purely model-free DRL approaches in real-world applications.

Purpose of the Study:

• To develop a hybrid algorithm integrating Model Predictive Control (MPC) with Deep Reinforcement Learning (DRL) to leverage the strengths of both approaches.
• To enable constraint satisfaction during the learning process in high-dimensional control tasks.
• To enhance sample efficiency and overall performance compared to existing DRL methods.

Main Methods:

• A novel hybrid algorithm, MPC-DRL, is constructed by combining MPC and DRL.
• The algorithm is designed to satisfy constraint conditions throughout the learning phase.
• The approach was validated on a reachability task with a static obstacle.

Main Results:

• The proposed MPC-DRL algorithm successfully satisfied the constraint condition (avoiding the static obstacle).
• The hybrid approach demonstrated improved sample efficiency, requiring fewer samples compared to state-of-the-art model-free algorithms.
• Higher overall performance was achieved with the MPC-DRL method.

Conclusions:

• The hybrid MPC-DRL algorithm effectively combines the benefits of MPC and DRL for efficient and constraint-aware learning.
• This approach offers a promising solution for complex control tasks requiring both high performance and adherence to constraints.
• The method shows potential for real-world applications where sample efficiency and safety are critical.