Model-Free Optimal Tracking Control via Critic-Only Q-Learning.

Area of Science:

• Control Theory
• Machine Learning
• Nonlinear Systems

Background:

• Model-free control is crucial for systems where dynamics are unknown.
• Optimal tracking control for nonaffine nonlinear discrete-time systems presents significant challenges.
• Existing methods often require complex system models or solving intricate equations.

Purpose of the Study:

• To develop a novel model-free optimal tracking control method for nonaffine nonlinear discrete-time systems.
• To introduce a critic-only Q-learning (CoQL) approach that avoids solving the tracking Hamilton-Jacobi-Bellman equation.
• To ensure the convergence and effectiveness of the proposed CoQL method, even with neural network approximation errors.

Main Methods:

• Development of a critic-only Q-learning (CoQL) algorithm using a single neural network for Q-function approximation.
• Establishment of Q-learning algorithm convergence based on an augmented system.
• Proof of CoQL method convergence considering neural network approximation errors.
• Design of adaptive optimal tracking control using a gradient descent scheme based on the learned Q-function.

Main Results:

• The proposed CoQL method successfully learns optimal tracking control policies from real system data.
• Convergence of the Q-learning algorithm and the CoQL method was theoretically established.
• Simulation studies demonstrated the effectiveness of the CoQL method in achieving optimal tracking control.
• The CoQL method was shown to be easy to implement and overcome exploration issues.

Conclusions:

• The critic-only Q-learning (CoQL) method provides an effective solution for model-free optimal tracking control of nonaffine nonlinear discrete-time systems.
• The CoQL approach simplifies implementation by using a critic-only structure and off-policy learning.
• The method addresses the challenge of inadequate exploration, making it a promising technique for real-world applications.