Counterfactual value decomposition for cooperative multi-agent reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multi-Agent Systems

Background:

• Value decomposition is crucial in Multi-Agent Reinforcement Learning (MARL).
• Traditional factored value functions (FVFs) struggle with non-monotonic payoffs.
• Existing methods approximate optimal joint actions, leading to biased value updates.

Purpose of the Study:

• To propose Comix, a novel off-policy MARL method addressing limitations in FVF updates.
• To enhance FVF update reliability by avoiding approximations of the optimal joint action value.
• To improve learning efficiency and performance in MARL tasks with non-monotonic payoffs.

Main Methods:

• Introduced the Sandwich Value Decomposition Framework for constrained FVF updates.
• Utilized orthogonal best responses to construct a reliable upper bound for FVF updates.
• Incorporated an attention mechanism for efficient and accurate upper bound computation.
• Ensured theoretical satisfaction of the Independent Gradient Maximization (IGM) property.

Main Results:

• Comix effectively constrains and guides FVF updates using both upper and lower bounds.
• The method overcomes the bias associated with approximating optimal joint action values.
• Achieved higher learning efficiency compared to state-of-the-art methods in experiments.
• Demonstrated superior performance on asymmetric One-Step Matrix Game, Predator-Prey, and StarCraft challenges.

Conclusions:

• Comix provides a more reliable approach to FVF updates in MARL.
• The proposed method enhances learning efficiency and performance, especially in complex scenarios.
• This framework offers a promising direction for advancing MARL research.