A Policy Gradient Algorithm to Alleviate the Multi-Agent Value Overestimation Problem in Complex Environments.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multi-agent reinforcement learning (MARL) is crucial for group decision-making in complex, high-dimensional environments.
• Existing deep policy gradient methods face challenges with reliability, stability, and convergence due to estimation errors and degraded experience quality.
• These limitations hinder performance in demanding applications like autonomous systems.

Purpose of the Study:

• To develop a novel MARL algorithm addressing the limitations of current deep policy gradient methods.
• To enhance the reliability, stability, and convergence of decision-making algorithms in complex state-action spaces.
• To improve the efficiency of experience sampling and overall algorithm performance.

Main Methods:

• Proposing the empirical clustering layer-based multi-agent dual dueling policy gradient (ECL-MAD3PG) algorithm.
• Integrating an empirical clustering layer to refine experience quality and sampling efficiency.
• Utilizing a dual dueling architecture to improve value estimation accuracy.

Main Results:

• The ECL-MAD3PG algorithm demonstrated superior performance across various complex environments.
• Achieved a significant 9.1% improvement in mission completion compared to the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm.
• Showcased enhanced reliability and stability in challenging scenarios, particularly in UAV cooperative combat.

Conclusions:

• ECL-MAD3PG effectively overcomes the convergence and stability issues of traditional MARL algorithms.
• The proposed algorithm offers a robust solution for complex, high-dimensional decision-making problems.
• ECL-MAD3PG shows significant promise for applications requiring reliable and adaptive multi-agent coordination.