Residual Q-Networks for Value Function Factorizing in Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Multiagent reinforcement learning (MARL) presents challenges in policy optimization due to increasing agent numbers.
• Existing methods like VDN, QMIX, QTRAN, and QPLEX use centralized training with decentralized execution (CTDE) but struggle with environmental complexity and stable convergence.
• Factorization of joint action-value functions is a key challenge in MARL.

Purpose of the Study:

• To introduce Residual Q-Networks (RQNs) as a novel approach to enhance MARL.
• To improve the robustness and stability of action-value function factorization in MARL.
• To accelerate convergence and achieve better performance in cooperative multiagent tasks.

Main Methods:

• Proposed Residual Q-Networks (RQNs) that learn to transform individual Q-value trajectories.
• RQNs preserve the individual-global-max (IGM) criteria while improving factorization robustness.
• RQNs function as auxiliary networks that become obsolete upon reaching training objectives.

Main Results:

• RQNs demonstrated faster convergence and increased stability compared to state-of-the-art methods (QPLEX, QMIX, QTRAN, VDN).
• The proposed method showed robust performance across a wider range of cooperative multiagent tasks.
• Improvements were particularly prominent in environments with high penalties for noncooperative behavior and incomplete state information during training.

Conclusions:

• Residual Q-Networks offer a significant advancement in MARL, addressing limitations of current factorization-based methods.
• RQNs provide a more stable and efficient training process for cooperative agents.
• The approach shows promise for complex MARL scenarios, including those with partial observability.