Off-policy reinforcement learning for H∞ control design.

Area of Science:

• Control Theory
• Nonlinear Systems
• Reinforcement Learning

Background:

• The nonlinear H∞ control problem requires solving the Hamilton-Jacobi-Isaacs (HJI) equation, a complex nonlinear partial differential equation.
• Analytical solutions for the HJI equation are generally impossible, and model-based methods fail with unknown or costly system models.

Purpose of the Study:

• To develop a novel method for solving the HJI equation for nonlinear H∞ control design when the system model is unknown.
• To enable robust control design using real system data rather than precise mathematical models.

Main Methods:

• An off-policy reinforcement learning (RL) approach is introduced to learn the HJI equation solution from system data.
• A neural network (NN)-based actor-critic architecture is employed for implementation.
• A least-square NN weight update algorithm is derived using the method of weighted residuals.

Main Results:

• The convergence of the proposed off-policy RL method for solving the HJI equation is theoretically proven.
• The method successfully learns the HJI equation solution using arbitrary policies for data generation, enhancing practical applicability.
• The NN-based off-policy RL method demonstrated effectiveness on a linear F16 aircraft plant and a rotational/translational actuator system.

Conclusions:

• The developed NN-based off-policy RL method offers a viable solution for nonlinear H∞ control design with unknown system models.
• This data-driven approach overcomes limitations of traditional model-based techniques, paving the way for more robust and practical control systems.