Graph based multi-agent reinforcement learning with evolutionary population for cooperation.

Area of Science:

• Artificial Intelligence
• Robotics
• Computer Science

Background:

• Existing Multi-Agent Reinforcement Learning (MARL) methods face challenges in scaling to complex coordination tasks due to limited agent observations and dynamic interactions.
• Convergence to optimal policies is difficult as task complexity and policy space increase, impacting stable policy evaluations.

Purpose of the Study:

• To propose GDE, a MARL framework designed to overcome scalability and convergence issues in cooperative multi-agent systems.
• To enhance agent coordination and information propagation in dynamic environments without requiring state consensus.

Main Methods:

• GDE integrates Graph-based value Decomposition with staged Evolutionary policy optimization.
• Evolutionary Algorithms (EAs) are utilized for gradient-free random search to improve policy exploration and convergence.
• Graph Neural Networks (GNNs) are employed to extend agent receptive fields and facilitate information propagation, leveraging permutation invariance for stable convergence with dynamic data.

Main Results:

• GDE demonstrates superior performance in complex coordination tasks, including StarCraft II micro-management, MAMuJoCo robot cooperation, and SUMO autonomous driving.
• The framework effectively captures complex coordination dynamics through multi-agent team formation and GNNs.
• Experimental results validate the effectiveness and necessity of each module within the GDE framework.

Conclusions:

• GDE offers a robust solution for enhancing coordination and policy convergence in MARL.
• The proposed combination of graph-based decomposition and evolutionary optimization is effective for complex multi-agent systems.
• The framework's modular design and adaptability make it suitable for diverse real-world applications.