Model-Based Chance-Constrained Reinforcement Learning via Separated Proportional-Integral Lagrangian

Area of Science:

• Artificial Intelligence
• Robotics
• Control Theory

Background:

• Real-world reinforcement learning (RL) demands robust safety guarantees under uncertainty.
• Existing chance-constrained RL methods often suffer from policy oscillations or suboptimal performance (overly conservative or unsafe).

Purpose of the Study:

• To develop an advanced algorithm that overcomes the limitations of current chance-constrained RL methods.
• To enhance the safety and reliability of RL policies in dynamic and uncertain environments.

Main Methods:

• Proposing a Separated Proportional-Integral Lagrangian (SPIL) algorithm, unifying proportional and integral control concepts.
• Employing an integral separation technique to manage integral values and stabilize learning.
• Utilizing model-based computation for the safe probability gradient to accelerate training.

Main Results:

• Demonstrated reduction in policy oscillations and conservatism in a car-following simulation.
• Successful application to a real-world mobile robot navigation task, ensuring obstacle avoidance despite unpredictable behaviors.

Conclusions:

• The SPIL algorithm offers a more stable and effective approach to chance-constrained reinforcement learning.
• SPIL enhances the practical applicability of RL in safety-critical real-world scenarios.