Optimal Learning Output Tracking Control: A Model-Free Policy Optimization Method With Convergence Analysis.

Area of Science:

• Intelligent Control
• Reinforcement Learning
• Control Systems

Background:

• Model-free optimal learning output tracking control (OLOTC) is gaining traction in intelligent control and reinforcement learning (RL).
• While off-policy learning and Q-learning have enabled model-free tracking control, direct policy learning (PL) methods remain underexplored despite their ease of implementation.

Purpose of the Study:

• To develop a novel model-free policy optimization (PO) algorithm for achieving OLOTC in unknown linear discrete-time (DT) systems.
• To address the gap in utilizing direct policy learning for model-free tracking control.

Main Methods:

• An iterative control policy is parameterized to directly enhance the augmented system's discounted value function using a gradient-based approach.
• A model-free two-point policy gradient (PG) algorithm is proposed to estimate the discounted value function's gradient from sampled states and reference trajectories.

Main Results:

• The proposed model-free PO algorithm demonstrates global convergence to the optimal value function under sufficient samples and appropriate conditions.
• Numerical simulations validate the effectiveness of the developed model-free PO algorithm for OLOTC.

Conclusions:

• The study successfully presents a novel model-free PO algorithm for OLOTC in unknown linear DT systems.
• The findings highlight the potential of direct policy learning approaches in model-free reinforcement learning for control applications.