HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target-Attacker-Defender Games

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Multi-Agent Systems

Background:

• Target-Attacker-Defender (TAD) games present complex multi-agent cooperative control challenges.
• Existing methods struggle with dynamic team sizes and energy constraints, leading to policy homogeneity and poor division of labor.

Purpose of the Study:

• To propose a Hierarchical Heterogeneous Multi-Agent Proximal Policy Optimization (HH-MAPPO) framework to address limitations in TAD pursuit-evasion games.
• To enable effective division of labor and differentiated behaviors in dynamic, energy-constrained multi-agent systems.

Main Methods:

• Developed a hierarchical framework with Role-Aware Embedding (RAE) for agents to learn unique roles and behaviors.
• Implemented an upper-level policy for target assignment and a lower-level policy for continuous control.
• Utilized a target-matching mechanism to manage dynamic observation spaces and maintain constant observation dimensions.

Main Results:

• HH-MAPPO demonstrated superior interception performance over baseline methods in various scenarios.
• Ablation studies showed RAE increased policy diversity, improving Sequence-Based Action Dissimilarity (SBAD) by 15.5%.
• Achieved a favorable performance-energy trade-off, maintaining a 70% interception rate under strict energy constraints.

Conclusions:

• The HH-MAPPO framework effectively resolves policy homogeneity and improves cooperative control in TAD games.
• Role-Aware Embedding is crucial for enabling diverse agent behaviors and efficient task allocation.
• The proposed method offers a robust solution for energy-constrained multi-agent systems facing dynamic challenges.