Robust Multi-Agent Reinforcement Learning by Mutual Information Regularization.

Area of Science:

• Artificial Intelligence
• Robotics
• Control Theory

Background:

• Cooperative multi-agent reinforcement learning (MARL) faces challenges in robustness due to unpredictable or adversarial agent actions.
• Existing robust MARL methods struggle with computational complexity and insufficient robustness as the number of agents increases.
• Human decision-making offers a model for robust behavior through general caution rather than exhaustive threat preparation.

Purpose of the Study:

• To develop a novel robust MARL method inspired by human decision-making to address computational and robustness limitations.
• To introduce Mutual Information Regularization as Robust Regularization (MIR3) for implicit worst-case robustness optimization.
• To enhance MARL agent caution and policy alignment with robust action priors.

Main Methods:

• Framing robust MARL as a control-as-inference problem.
• Employing off-policy evaluation for implicit optimization of worst-case robustness.
• Introducing MIR3, a mutual information regularization technique, to maximize a lower bound on robustness during training.

Main Results:

• MIR3 significantly surpasses baseline methods in robustness and training efficiency in MARL.
• The method maintains cooperative performance in complex simulations like StarCraft II and swarm control tasks.
• Real-world deployment of MIR3 in robot swarm control yielded a 14.29% reward improvement over the best baseline.

Conclusions:

• MIR3 provides an effective and efficient approach to achieve robust MARL by acting as an information bottleneck and promoting cautious agent behavior.
• The proposed method offers a scalable solution for real-world MARL applications requiring resilience against adversarial conditions.
• MIR3 demonstrates superior performance and robustness compared to existing methods in diverse cooperative MARL scenarios.