Solver-Critic: A Reinforcement Learning Method for Discrete-Time-Constrained-Input Systems.

Area of Science:

• Control Engineering
• Artificial Intelligence
• Optimization Theory

Background:

• Optimal control problems for discrete-time systems with input constraints are challenging.
• Existing methods often require complex reward functions or struggle with convergence.

Purpose of the Study:

• To develop a novel solver-critic (SC) architecture for discrete-time (DT) constrained-input systems.
• To address the limitations of current approaches in handling input constraints within optimal control.

Main Methods:

• The proposed SC architecture comprises a critic network, an action solver, and a target network.
• The critic network approximates the action-value function using sum-of-squares (SOS) polynomials.
• The action solver utilizes SOS programming for control input generation within constraints, and a target network stabilizes learning.

Main Results:

• The SC architecture effectively solves constrained-input control problems without non-quadratic reward functionals.
• Theoretical analysis confirms the convergence property of the learned policy to the Hamilton-Jacobi-Bellman equation's optimal solution.
• Numerical examples validate the approach's effectiveness and theoretical underpinnings.

Conclusions:

• The developed SC architecture provides an effective solution for optimal control of discrete-time systems with input constraints.
• The method demonstrates robust convergence and practical applicability validated through simulations.
• This work contributes a stable and efficient approach to a significant control engineering challenge.