MACRPO: Multi-agent cooperative recurrent policy optimization

Area of Science:

• Artificial Intelligence
• Reinforcement Learning
• Multi-Agent Systems

Background:

• Multi-agent systems face challenges in partially observable and non-stationary environments without communication.
• Effective information sharing and coordination are crucial for cooperative policy learning.

Purpose of the Study:

• To propose a novel multi-agent actor-critic method, MACRPO, for enhanced cooperative policy learning.
• To improve information integration across agents and time in challenging multi-agent settings.

Main Methods:

• Implemented a recurrent layer in the critic's network architecture with a meta-trajectory training framework.
• Developed a novel advantage function incorporating other agents' rewards and value functions, controlled by a cooperation parameter.
• Evaluated MACRPO on Deepdrive-Zero, Multi-Walker, and Particle environments.

Main Results:

• MACRPO demonstrated superior performance compared to state-of-the-art multi-agent algorithms and single-agent methods.
• The recurrent layer effectively learned cooperation, agent interactions, and handled partial observability.
• The proposed advantage function allowed control over the level of cooperation.

Conclusions:

• MACRPO offers a robust solution for learning cooperative policies in challenging multi-agent scenarios.
• The method's novel components effectively address partial observability and facilitate inter-agent coordination.
• The findings suggest MACRPO's potential for advancing research in cooperative multi-agent reinforcement learning.