Learning Multi-Agent Cooperation via Considering Actions of Teammates

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multi-Agent Systems

Background:

• Centralized Training with Decentralized Execution (CTDE) methods, like QMIX, excel in cooperative multi-agent reinforcement learning (MARL).
• QMIX's monotonic constraint limits its applicability to non-monotonic reward structures.
• Existing MARL methods struggle with generalization to unseen environments and varying team compositions (ad hoc team play).

Purpose of the Study:

• To propose a novel Q-value decomposition addressing non-monotonicity in MARL.
• To develop an adaptive agent strategy for ad hoc team play.
• To enhance the performance and generalization capabilities of CTDE MARL methods.

Main Methods:

• A novel Q-value decomposition considering individual and cooperative returns.
• A greedy action-searching method robust to agent configuration changes.
• Integration of an environmental cognition consistency loss and a modified prioritized experience replay (PER) buffer.

Main Results:

• Significant performance improvements in both monotonic and non-monotonic cooperative tasks.
• Successful adaptation to ad hoc team play situations with varying agents and action orders.
• Demonstrated superior exploration and robustness compared to existing MARL methods.

Conclusions:

• The proposed Q-value decomposition effectively handles non-monotonicity in MARL.
• The developed method achieves robust generalization and excels in ad hoc team play.
• This work advances CTDE MARL by improving adaptability and performance in complex cooperative environments.