Offline constrained policy optimization with safe anchoring.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Real-world reinforcement learning (RL) deployment is hindered by safety risks and distribution shift in offline settings.
• Existing methods struggle to balance performance optimization with critical safety constraints.

Purpose of the Study:

• To develop a safe offline reinforcement learning algorithm that guarantees performance improvement and bounds costs.
• To introduce a novel 'safe anchoring' mechanism to prevent out-of-distribution actions and ensure constraint satisfaction.

Main Methods:

• Formulated safe offline RL as a constrained policy optimization problem using Lagrangian duality.
• Developed the Offline Constrained Policy Optimization with Safe Anchoring (OCPO-SA) algorithm, integrating a VAE-distilled safe anchoring mechanism.
• Ensured monotonic performance improvement and bounded worst-case costs relative to the behavioral policy.

Main Results:

• OCPO-SA demonstrated guaranteed safety across all tested environments in Safety-Gymnasium and Bullet-Safety-Gym.
• The algorithm achieved a 24% average cost reduction compared to the best baseline.
• The safe anchoring mechanism effectively prevented out-of-distribution actions, enhancing constraint adherence.

Conclusions:

• OCPO-SA offers a robust solution for safe offline reinforcement learning, addressing key limitations of current approaches.
• The proposed method provides a practical framework for deploying RL in safety-critical real-world applications.
• Safe anchoring is a crucial component for maintaining safety and stability in offline RL.