A Collaborative Multiagent Reinforcement Learning Method Based on Policy Gradient Potential

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Gradient-based methods are prevalent in multi-agent reinforcement learning (MARL).
• Limited research exists on the convergence of existing MARL algorithms for identical interest games.
• Agents often lack access to payoff matrices and joint strategies in real-world scenarios.

Purpose of the Study:

• To propose a novel Policy Gradient Potential (PGP) algorithm for MARL.
• To address the convergence challenges in identical interest games.
• To enable learning optimal joint strategies using only local information and reward probabilities.

Main Methods:

• Developed the Policy Gradient Potential (PGP) algorithm, utilizing PGP for strategy updates instead of direct gradients.
• Defined the performance index as the probability of achieving maximal reward, suitable for scenarios with limited information.
• Conducted theoretical analysis on a continuous model of identical interest repeated games.
• Performed experimental comparisons against other MARL algorithms on collaborative tasks and a real-world navigation problem.

Main Results:

• Theoretical analysis indicates asymptotic stability for optimal joint actions under unique component action conditions.
• Experimental results demonstrate superior performance of the PGP algorithm compared to other MARL methods.
• PGP achieved higher cumulative rewards and reduced time steps in tested scenarios.

Conclusions:

• The PGP algorithm offers a robust and effective approach for MARL in identical interest games.
• It successfully addresses limitations of traditional gradient-based methods, especially with incomplete information.
• PGP shows significant potential for real-world applications requiring decentralized learning and coordination.