Nonconvex Policy Search Using Variational Inequalities.

Area of Science:

• Robotics
• Control Theory
• Machine Learning

Background:

• Policy search methods in reinforcement learning are effective for high-dimensional control problems like robotics.
• Current safe policy search methods are limited to convex constraints, risking hardware damage with unsafe parameters.
• Existing approaches lack the ability to manage nonconvex policy constraints, a common issue in real-world applications.

Purpose of the Study:

• To develop the first safe policy search reinforcement learning algorithm capable of handling nonconvex policy constraints.
• To establish a novel connection between nonconvex variational inequalities and policy search problems.
• To introduce algorithms that ensure safety in reinforcement learning policy optimization.

Main Methods:

• Proposing projection-based methods for safe policies under nonconvex constraints.
• Leveraging a newly discovered connection between nonconvex variational inequalities and policy search.
• Developing and analyzing two algorithms: Mann and two-step iteration for solving these problems.
• Proving convergence guarantees for the proposed algorithms in the nonconvex stochastic setting.

Main Results:

• Demonstrated the first safe policy search reinforcement learner for nonconvex constraints.
• Successfully applied Mann and two-step iteration algorithms, proving their convergence.
• Outperformed previous methods on six benchmark dynamical systems under various settings.
• Validated the capability of the new methods to ensure safe policy parameters and prevent hardware damage.

Conclusions:

• The developed algorithms provide a robust solution for safe policy search under nonconvex constraints.
• This work significantly advances reinforcement learning safety in complex control problems.
• The proposed methods offer a practical and effective approach for real-world robotics and control applications.