Efficient Heterogeneous Exploration with Mutual Policy Divergence Maximization for Multiagent Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Multi-Agent Systems

Background:

• Multi-Agent Reinforcement Learning (MARL) excels in cooperative tasks but struggles with generalization in heterogeneous scenarios requiring agent specialization.
• Existing MARL methods often disable parameter sharing, yet this alone is insufficient for effective behavioral specialization.
• Sequential updating schemes improve specialization but lack tailored exploration strategies for leveraging inter-agent information flow.

Purpose of the Study:

• To enhance exploration efficiency and policy heterogeneity in sequential MARL settings.
• To introduce a novel MARL framework, Multi-Agent Divergence Policy Optimization (MADPO), addressing limitations in current sequential updating methods.
• To develop and validate new divergence measures for guiding exploration and promoting behavioral heterogeneity.

Main Methods:

• Proposed Multi-Agent Divergence Policy Optimization (MADPO) framework featuring Mutual Policy Divergence Maximization (Mutual PDM).
• Mutual PDM incorporates intra-agent PDM (encouraging novel behaviors) and inter-agent PDM (promoting heterogeneity with predecessors).
• Employed Conditional Cauchy-Schwarz (CCS) divergence and introduced Generalized Conditional Cauchy-Schwarz (GCCS) divergence for stable, entropy-guided exploration and heterogenization.

Main Results:

• Theoretically proved that CCS and GCCS divergences provide stable, entropy-guided incentives for exploration and heterogenization.
• Empirically demonstrated a strong correlation between overall performance and policy heterogeneity in heterogeneous MARL.
• MADPO consistently outperformed state-of-the-art sequential updating methods across three challenging MARL benchmarks with over 25 heterogeneous scenarios.

Conclusions:

• MADPO effectively enhances both exploration efficiency and policy heterogeneity in sequential MARL.
• The proposed CCS and GCCS divergences offer a theoretically sound and empirically validated approach to guide MARL agents.
• The study highlights the critical role of policy heterogeneity for achieving superior performance in complex, heterogeneous MARL tasks.