Robust Reward-Free Actor-Critic for Cooperative Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Cooperative multiagent reinforcement learning (MARL) faces challenges with diverse, private reward functions.
• Centralized training with decentralized execution (CTDE) is a common paradigm.
• Byzantine attacks, where malicious agents send false information, threaten system integrity.

Purpose of the Study:

• To develop a privacy-preserving MARL algorithm for CTDE with diverse rewards.
• To create a robust MARL algorithm resilient to Byzantine attacks.
• To introduce novel Byzantine attacks and robust aggregation rules specific to reinforcement learning.

Main Methods:

• Proposed a reward-free deep deterministic policy gradient (RF-DDPG) algorithm, transmitting critic gradients instead of rewards for privacy.
• Developed a robust extension, R2F-DDPG, employing robust aggregation rules to counter Byzantine attacks.
• Introduced projection-boosted robust aggregation rules to address novel RL-specific Byzantine attacks.

Main Results:

• RF-DDPG successfully enabled cooperative training of agents without Byzantine threats.
• R2F-DDPG demonstrated significant robustness against sophisticated Byzantine attacks.
• The proposed robust aggregation rules effectively mitigated the impact of RL-specific attacks.

Conclusions:

• The RF-DDPG algorithm offers a privacy-preserving solution for cooperative MARL.
• R2F-DDPG provides a robust framework for MARL systems susceptible to Byzantine adversaries.
• This work advances the security and privacy of multiagent reinforcement learning systems.