Action Mapping: A Reinforcement Learning Method for Constrained-Input Systems.

Area of Science:

• Control Systems Engineering
• Artificial Intelligence
• Optimization Theory

Background:

• Existing optimal control methods primarily address input saturation, neglecting other critical constraints like inequalities and state-dependencies.
• Constrained-input optimal control is crucial for real-world discrete-time systems but faces limitations with diverse constraint types.

Purpose of the Study:

• To develop a reinforcement learning-based algorithm for discrete-time systems with general input constraints.
• To address the limitations of existing methods in handling combined inequality and state-dependent constraints.

Main Methods:

• A reinforcement learning (RL) algorithm utilizing the deterministic policy gradient (DPG) to solve the Hamilton-Jacobi-Bellman (HJB) equation.
• An action mapping (AM) mechanism is proposed to transform the constrained exploration space into a standard Cartesian product space for effective searching.

Main Results:

• The proposed algorithm successfully learns policies that generate control signals satisfying complex input constraints.
• Convergence analysis demonstrates the algorithm's convergence to the optimal solution of the HJB equation.
• The continuity of the iterative estimated Q-function was investigated and validated.

Conclusions:

• The developed RL-based approach effectively handles constrained-input optimal control problems in discrete-time systems.
• The action mapping mechanism provides a robust way to manage diverse input constraints without altering the reward function.
• Numerical examples confirm the algorithm's practical effectiveness and theoretical convergence properties.