Safe Exploration Algorithms for Reinforcement Learning Controllers

Area of Science:

• Robotics and Autonomous Systems
• Control Theory
• Machine Learning

Background:

• Self-learning, particularly reinforcement learning, enables autonomous control of complex systems.
• Exploration in unknown or dangerous environments poses significant safety challenges for learning agents.
• Existing methods often require global safety functions or detailed system dynamics, limiting their applicability.

Purpose of the Study:

• To develop a novel approach for safe exploration in uncertain environments for autonomous agents.
• To overcome limitations of existing methods by avoiding the need for global safety functions or specific system dynamics.
• To enable learning agents to handle dangerous states with limited perception capabilities.

Main Methods:

• Interval estimation of agent dynamics during exploration.
• Development of the Safety Handling Exploration with Risk Perception Algorithm (SHERPA) using temporary safety functions (backups).
• Development of OptiSHERPA, an advanced algorithm utilizing safety metrics for more complex systems.

Main Results:

• SHERPA successfully avoided dangerous states in a simulated quadrotor control task.
• OptiSHERPA demonstrated safe handling of a more dynamically complex aircraft altitude control task.
• The proposed approach effectively manages exploration risks without prior knowledge of global safety or system dynamics.

Conclusions:

• The interval estimation approach provides a robust framework for safe exploration in autonomous systems.
• SHERPA and OptiSHERPA offer practical solutions for real-world applications requiring safe autonomous control.
• This work advances the field of reinforcement learning by addressing critical safety concerns during exploration.