SMIX(λ): Enhancing Centralized Value Functions for Cooperative Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Centralized Value Function (CVF) learning in Multiagent Reinforcement Learning (MARL) is challenging due to the exponentially increasing joint action space.
• Existing methods often rely on greedy assumptions, limiting stability and generalizability.

Purpose of the Study:

• To propose SMIX(λ), an novel approach for stable and generalizable CVF learning in MARL.
• To overcome the computational cost and numerical instability of importance sampling in off-policy MARL training.

Main Methods:

• SMIX(λ) utilizes off-policy training with λ-returns as a proxy for temporal difference (TD) error computation.
• A modified QMIX network architecture serves as the base, integrated with the Q(λ) approach from an expectation correction perspective.

Main Results:

• SMIX(λ) demonstrates equivalence to Q(λ), inheriting its convergence properties without the curse of dimensionality.
• Experiments on the StarCraft Multiagent Challenge (SMAC) show SMIX(λ) significantly outperforms state-of-the-art MARL methods.

Conclusions:

• SMIX(λ) offers a stable and scalable solution for CVF learning in MARL.
• The method can be broadly applied to enhance other centralized training with decentralized execution (CTDE) algorithms by improving their CVFs.