CVaR-Constrained Policy Optimization for Safe Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Constrained reinforcement learning (RL) often guarantees safety only in expectation.
• This expectation-based safety is insufficient for safety-critical applications, risking high-probability constraint violations.
• Ensuring high probabilities of constraint satisfaction is crucial for safe RL.

Purpose of the Study:

• To develop a novel algorithm for safe reinforcement learning that addresses limitations of expectation-based safety guarantees.
• To maximize expected return while ensuring high probabilities of satisfying safety constraints.
• To focus on the upper tail of constraint costs for robust safety.

Main Methods:

• Propose the CVaR-constrained policy optimization (CVaR-CPO) algorithm.
• Formulate the problem in an augmented state space using state extension and trust-region methods.
• Apply the Lagrangian method for policy updates and utilize quantile-based estimation for CVaR-related value functions.

Main Results:

• CVaR-CPO effectively maximizes expected return while adhering to safety constraints.
• The method demonstrates high probabilities of constraint satisfaction in experiments.
• Performance is comparable to existing state-of-the-art safe RL methods.

Conclusions:

• CVaR-CPO offers a robust approach to safe reinforcement learning by directly addressing conditional value-at-risk (CVaR).
• The algorithm provides a practical solution for safety-critical decision-making problems.
• This work advances the field of safe RL by improving constraint satisfaction guarantees.