A Distributional Perspective on Multiagent Cooperation With Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Multiagent Reinforcement Learning
• Game Theory

Background:

• Current value decomposition methods in multiagent reinforcement learning (MARL) use a scalar global Q value, which oversimplifies individual agent dynamics and correlations.
• This simplification limits the accurate modeling of distributed system dynamics, hindering optimal performance in cooperative multiagent settings.

Purpose of the Study:

• To propose a novel distributional framework, Distributional Multiagent Cooperation (DMAC), to explicitly model correlations between cooperative agents.
• To enhance the performance of multiagent systems by accurately representing distributed agent dynamics.

Main Methods:

• DMAC treats individual agent Q values as value distributions, with expectations representing overall system performance.
• Employs distributional reinforcement learning to minimize the difference between estimated and target value distributions for optimization.
• Evaluates DMAC across nine diverse scenarios within the StarCraft Multiagent Challenge (SMAC).

Main Results:

• DMAC explicitly models the distributed dynamics of agents, leading to improved performance.
• Extensive experiments demonstrate DMAC's significant outperformance compared to baseline methods.
• Achieved superior average median test win rates across various SMAC scenarios.

Conclusions:

• The proposed distributional framework (DMAC) offers a more effective approach to MARL by capturing agent correlations.
• DMAC's ability to model distributed dynamics leads to enhanced cooperative multiagent system performance.
• This research provides a promising direction for advancing MARL algorithms.