Asynchronous iterative Q-learning based tracking control for nonlinear discrete-time multi-agent systems

Area of Science:

• Control Systems Engineering
• Artificial Intelligence
• Robotics

Background:

• Multi-agent systems (MASs) present complex challenges in coordinated control.
• Tracking control is crucial for MASs to follow desired trajectories.
• Existing algorithms may lack efficiency in nonlinear discrete-time MASs.

Purpose of the Study:

• To develop a novel tracking control algorithm for nonlinear discrete-time MASs.
• To transform the tracking problem into an optimal regulation problem of a local neighborhood error system (LNES).
• To implement the algorithm using a neural network-based actor-critic framework.

Main Methods:

• Construction of a local neighborhood error system (LNES).
• Development of an asynchronous iterative Q-learning (AIQL) algorithm with two Q-values (Q_i^A and Q_i^B) for policy improvement and evaluation.
• Implementation of AIQL using a neural network-based actor-critic framework with two critic networks.

Main Results:

• The LNES is shown to converge to 0, effectively solving the tracking problem.
• The AIQL-based algorithm achieved a lower cost value compared to the iterative Q-learning (IQL) based algorithm.
• The AIQL algorithm exhibited a faster convergence speed than the IQL-based algorithm.

Conclusions:

• The proposed AIQL algorithm provides an effective solution for tracking control in nonlinear discrete-time MASs.
• The AIQL algorithm offers improved performance in terms of cost and convergence speed.
• The neural network-based actor-critic implementation successfully demonstrates the algorithm's capabilities.