HH-MAPPO: A Hierarchical Reinforcement Learning Framework for Dynamic-Scale Target-Attacker-Defender Games

The Target-Attacker-Defender (TAD) pursuit-evasion game is a core challenge in multi-agent cooperative control, yet real-world settings involving dynamic team scaling and strict energy constraints remain largely unaddressed. When scalable shared-parameter policies are adopted to cope with the varying number of agents, severe policy homogeneity emerges, preventing effective division of labor. This paper proposes a Hierarchical Heterogeneous Multi-Agent Proximal Policy Optimization (HH-MAPPO) framework to resolve these challenges. Both levels employ actor-critic networks with Role-Aware Embedding (RAE). In this mechanism, each agent is assigned a unique, learnable role embedding derived from its identity. These embeddings serve as conditioning inputs to the shared policy network, enabling it to generate differentiated behaviors and effectively mitigating policy homogeneity. The upper-level policy determines the number of defenders to deploy and assigns interception targets, while the lower-level policy handles continuous control of each defender and the ground moving target (GMT). This hierarchy resolves dynamic observation spaces via a target-matching mechanism, where each defender's observation includes only its own state and its assigned attacker's state, keeping observation dimension constant. Experiments in a 3D TAD simulation with continuous attacker arrivals and energy-constrained defenders show the following: (1) HH-MAPPO achieves superior interception performance compared to baseline methods in both symmetric and asymmetric scenarios; (2) ablation studies confirm RAE increases policy diversity, raising Sequence-Based Action Dissimilarity (SBAD) by 15.5%; and (3) Pareto analysis demonstrates a superior performance-energy trade-off, maintaining about 70% interception rate even under an extreme energy cap (E = 30).