Model-Free Q-Learning for the Tracking Problem of Linear Discrete-Time Systems

Area of Science:

• Control Systems Engineering
• Machine Learning
• Robotics

Background:

• Tracking problems in linear discrete-time systems often suffer from unknown system dynamics, hindering precise control.
• Existing adaptive dynamic programming (ADP) and Q-learning methods have limitations in handling completely unknown system parameters for tracking tasks.

Purpose of the Study:

• To propose a model-free Q-learning algorithm for solving the tracking problem in linear discrete-time systems with unknown dynamics.
• To develop a novel performance index that transforms the tracking problem into a regulation problem, thereby eliminating tracking errors.

Main Methods:

• A model-free Q-learning algorithm is introduced, utilizing an enhanced performance index with an added product term.
• The control policy is deduced iteratively using online system state, control input, and reference trajectory information, without prior system knowledge.
• An off-policy approach is incorporated to optimize data usage for deriving the optimal control policy.

Main Results:

• The proposed performance index effectively converts the tracking problem into a regulation problem, aiming for zero steady-state error.
• The iterative Q-learning method, combined with the new performance index, yields a control policy that theoretically eliminates tracking errors.
• Numerical simulations validate the effectiveness of the proposed algorithm in achieving accurate system tracking.

Conclusions:

• The developed model-free Q-learning algorithm offers a robust solution for tracking control in systems with unknown dynamics.
• The novel performance index and iterative update strategy ensure accurate tracking and efficient policy learning.
• This approach provides a significant advancement over existing methods, particularly in scenarios with limited system information.