GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Safe reinforcement learning (SRL) integrates safety constraints into deep reinforcement learning (DRL).
• SRL efficacy is limited by data insufficiency in early learning stages, hindering accurate function approximation.
• Existing SRL methods struggle with reliable safety during initial training phases.

Purpose of the Study:

• Introduce GenSafe, a generalizable safety enhancer for SRL.
• Overcome data insufficiency challenges in early SRL.
• Enhance the safety performance and constraint satisfaction of DRL agents.

Main Methods:

• Leverage model order reduction techniques to construct a reduced order Markov decision process (ROMDP).
• Utilize ROMDP as a low-dimensional approximator for original safety constraints.
• Refine agent actions by solving ROMDP-based constraints to improve safety.

Main Results:

• GenSafe significantly improves safety performance, particularly in the early learning phases of SRL.
• The approach effectively addresses data insufficiency issues common in initial training.
• Maintained satisfactory task performance alongside enhanced safety.

Conclusions:

• GenSafe acts as an effective additional safety layer for SRL algorithms.
• The method demonstrates broad compatibility with various SRL approaches and problems.
• GenSafe offers a novel solution for augmenting existing SRL methods to improve safety and learning efficiency.