Multistep Q-Learning-Based Optimal Consensus Control of Linear Discrete-Time Multiagent Systems

This article considers the optimal consensus control for the multiagent systems problem. By developing the multiagent multistep Q-learning (MaMsQL), the methodology achieves enhanced efficiency while addressing the issue of the complex interaction dynamics between agents, environmental uncertainty, thus ultimately meeting demand of balancing exploration and exploitation. First, associated with the performance index, the Q-function is established to prove that all optimal Q-functions form a Nash equilibrium outcome, thereby the consensus problem is converted to finding the optimal Q-functions. Then, the MaMsQL method is developed with theoretical proof of its convergence. Finally, the method is implemented through a specially designed Actor-Critic network. By virtue of the comparison with multiagent single step Q-learning, the effectiveness and superiority of this method are verified through simulation examples.