Mirror Descent Safe Policy Optimization for Reinforcement Learning Agents.

Area of Science:

• Artificial Intelligence
• Reinforcement Learning
• Robotics

Background:

• Embodied agents require mechanisms for complex problem-solving.
• Reinforcement learning (RL) is a key AI technology for enhancing agent learning capabilities.
• Safe exploration is critical as not all actions are acceptable.

Purpose of the Study:

• To propose a novel algorithm, Mirror Descent Safe Policy Optimization (MDSPO), for safe reinforcement learning.
• To ensure RL agents maximize returns while adhering to safety constraints during exploration.

Main Methods:

• Developed a novel optimization objective for safe policy optimization.
• Employed a three-stage optimization strategy: unconstrained gradient descent, nonparametric policy projection with cost constraints, and parametric policy projection.
• Utilized mirror descent optimization to balance return maximization and safety.

Main Results:

• MDSPO improves average return by approximately 12% in locomotive experiments.
• Demonstrated superior satisfaction of cost constraints compared to state-of-the-art methods.
• Successfully found optimal paths and guaranteed agent safety in a real-world unmanned surface vessel obstacle avoidance task.

Conclusions:

• MDSPO is a simple, first-order, and easily implementable safe RL algorithm.
• Theoretical analysis provides bounds on return improvement and constraint violation.
• MDSPO effectively enhances agent performance and safety in constrained environments.