Accelerated Value Iteration-Based Safe Q-Learning for Data-Driven Optimal Tracking Control.

Area of Science:

• Control Systems Engineering
• Machine Learning
• Nonlinear Dynamics

Background:

• Designing tracking controllers for unknown nonlinear systems presents challenges in ensuring both performance and safety.
• Existing methods may struggle with optimality, convergence speed, or maintaining system constraints.

Purpose of the Study:

• To develop an accelerated Safe Q-Learning (SQL) algorithm for designing tracking controllers in unknown nonlinear systems.
• To ensure both the optimality and safety of the tracking controller through a novel Q-function design.
• To accelerate the learning process and reduce computational load, especially with large datasets.

Main Methods:

• An augmented Q-function combining a quadratic utility function (for optimality) and a control barrier function (CBF, for safety) was devised.
• An accelerated iterative learning mechanism, including policy evaluation (PE) and policy improvement (PI), was employed.
• The PE process integrated Q-function differences for faster convergence, while PI utilized Nesterov Momentum for acceleration.

Main Results:

• The proposed augmented Q-function ensures tracking error elimination and faster convergence within safe boundaries.
• The accelerated iterative learning mechanism effectively reduces computational pressure when handling large offline datasets.
• Theoretical analysis confirmed the convergence of the Q-function sequence and the safety of the derived optimal tracking policy.

Conclusions:

• The accelerated SQL algorithm provides an effective approach for designing safe and optimal tracking controllers for unknown nonlinear systems.
• The integration of CBF and accelerated learning techniques offers significant advantages in performance and computational efficiency.
• Simulation examples using neural networks and an actor-critic structure validated the practical availability of the developed methods.