Sparse Communication for Policy Shaping in Multi-Agent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multi-agent reinforcement learning (MARL) faces challenges in efficient coordination with limited communication.
• Existing MARL methods often overlook how communication impacts policy learning, causing redundant interactions.

Purpose of the Study:

• To propose a threshold-gated sparse communication framework for MARL to improve coordination and reduce communication overhead.
• To integrate communication directly into agent utility functions for enhanced policy learning.

Main Methods:

• Developed a sparse communication framework building upon the QMIX value-decomposition method.
• Agents encode observations, use a learned trigger for communication, and aggregate messages via neighbor-constrained attention.
• Communication is incorporated into utility estimation for decentralized decision-making.

Main Results:

• Achieved significant reductions in communication frequency (e.g., 30-38% in MMM, 28-37% in 10m_vs_11m) on the StarCraft Multi-Agent Challenge (SMAC) benchmark.
• Improved win rates compared to QMIX (e.g., 96.6% vs 92.1% in MMM, 88.4% vs 85.6% in 10m_vs_11m).
• Demonstrated that sparse communication regulates policy formation and enhances coordination quality and training stability.

Conclusions:

• Selective communication in MARL enables efficient coordination and reduces computational overhead.
• The proposed framework effectively regulates policy formation and improves decentralized decision-making.
• Sparse communication is crucial for shaping effective coordination policies in MARL systems.