Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks

Area of Science:

• Control Theory
• Machine Learning
• Artificial Intelligence

Background:

• Optimal control of unknown systems is challenging.
• Traditional methods often require full system dynamics knowledge.
• Online learning offers a path for adaptive control strategies.

Purpose of the Study:

• To develop an online policy iteration (PI) algorithm for continuous-time optimal control.
• To address unknown constrained-input systems without prior dynamics knowledge.
• To ensure stability and convergence of the adaptive control system.

Main Methods:

• An actor-critic structure with two neural networks (NNs) for control policy generation.
• A novel NN identifier to obviate the need for system dynamics knowledge.
• A new learning rule for exponential convergence of identifier weights and experience replay for excitation.
• Simultaneous adaptation of actor, critic, and identifier networks.

Main Results:

• Guaranteed stability of the entire adaptive system (actor, critic, system state, identifier).
• Convergence of identifier weights to small neighborhoods of ideal values.
• Demonstrated convergence to a near-optimal control law.
• Effectiveness validated through a simulation example.

Conclusions:

• The proposed online PI algorithm effectively learns optimal control for unknown systems.
• The NN identifier and novel learning rule ensure robust and stable adaptation.
• Experience replay enhances learning efficiency and excitation.
• The method provides a practical approach to optimal control in complex, unknown environments.