Learning Optimal Policies With Local Observations for Cooperative Multiagent Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multiagent Systems

Background:

• Cooperative multiagent reinforcement learning (MARL) faces a core dilemma: balancing reward maximization (exploitation) with information gathering (exploration) under partial observability.
• Existing methods often combine exploration and exploitation suboptimally, leading to task failures.

Purpose of the Study:

• To theoretically prove the existence of a latent state that ensures optimal individual and global policies in MARL.
• To develop a novel method that unifies exploration and exploitation within a single framework for improved MARL performance.

Main Methods:

• Proposed a weighted value function factorization approach named unified MARL (UMARL).
• Introduced agent representation networks (ARNs) and individual weighting networks (IWNs) for learning unified agent representations and credit assignment.
• Designed a latent state regularizer (LSR) to approximate the theoretically derived latent state using local observations.

Main Results:

• Theoretically proved that a latent state exists and can be approximated from local observations.
• UMARL demonstrated superior performance compared to 12 state-of-the-art methods across diverse MARL benchmarks.
• Achieved significant improvements in the m-step matrix game, Level-Based Foraging (LBF), StarCraft II, and Google Research Football (GRF).

Conclusions:

• UMARL effectively unifies exploration and exploitation in MARL by leveraging a latent state representation.
• The proposed method offers a more robust and optimal solution to the exploration-exploitation dilemma in partially observable MARL settings.
• UMARL represents a significant advancement in cooperative MARL, enhancing performance in complex, real-world applications.