Optimal Containment of Multiagent Systems With Multistep Policy Gradient Reinforcement Learning

Area of Science:

• Control Theory
• Artificial Intelligence
• Machine Learning

Background:

• Multiagent systems (MASs) present complex control challenges, especially with unknown dynamics.
• Optimal containment control is crucial for coordinating MAS behavior.

Purpose of the Study:

• To develop an efficient algorithm for optimal containment control in discrete-time MASs.
• To address challenges of unknown system dynamics and asynchronous agent updates.

Main Methods:

• Integration of multistep temporal difference (TD) learning with policy gradient (PG) reinforcement learning (RL) to form an online off-policy multistep PG (MS-PG) algorithm.
• Utilizing an actor-critic neural network (NN) architecture for policy and Q-function approximation.
• Incorporating an experience replay (ER) mechanism for enhanced training and sample efficiency.

Main Results:

• The proposed MS-PG algorithm achieves optimal control performance in MASs with unknown dynamics.
• Rigorous establishment of closed-loop system stability and algorithmic convergence.
• Demonstrated effectiveness of the hybrid learning framework combining offline and online data.

Conclusions:

• The MS-PG algorithm offers a robust and efficient solution for optimal containment control in MASs.
• The actor-critic NN architecture and ER mechanism significantly improve learning efficiency.
• Simulation results validate the practical applicability and effectiveness of the proposed method.