Learn Zero-Constraint-Violation Safe Policy in Model-Free Constrained Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Model-free reinforcement learning (RL) often struggles with safety constraints, as traditional methods require experiencing dangerous situations to learn avoidance.
• Existing posterior penalty methods in RL cannot guarantee zero constraint violations, even after convergence, limiting their applicability in safety-critical domains.

Purpose of the Study:

• To develop a model-free reinforcement learning approach capable of learning zero-constraint-violation safe policies.
• To propose a novel algorithm, the Safe Set Actor-Critic (SSAC), that leverages energy functions for enhanced safety.

Main Methods:

• Utilizing data-driven methods to learn safety-oriented energy functions, eliminating the need for known environment dynamics.
• Formulating a constrained reinforcement learning problem to derive and optimize zero-violation policies using Lagrangian-based methods.

Main Results:

• The proposed SSAC algorithm successfully learns policies that achieve zero-constraint violation across complex simulation environments.
• Experimental validation, including a hardware-in-loop test with an autonomous vehicle controller, demonstrates the efficacy of the approach.
• The learned policies exhibit performance comparable to model-based baselines while ensuring absolute safety.

Conclusions:

• The SSAC algorithm effectively addresses the challenge of learning zero-constraint-violation safe policies in model-free RL.
• The integration of energy functions and constrained RL provides a robust framework for safe decision-making in autonomous systems.
• This work paves the way for safer and more reliable RL applications in real-world scenarios.