Entropy-Regularized Hierarchical MARL for Resilient Moving Target Defense in Cyber-Physical Systems

Cyber-Physical Systems (CPS), including smart grids and industrial control networks, must maintain secure and stable operations despite increasingly adaptive cyber threats. Existing moving target defense (MTD) approaches often rely on fixed reconfiguration strategies or flat learning architectures that fail to scale and do not explicitly ensure operational resilience under real-time constraints. This study proposes a resilience-oriented hierarchical multi-agent reinforcement learning (MARL) framework for adaptive MTD in CPS environments. The attacker-defender interaction is modeled as a partially observable stochastic game, enabling defenders to learn adaptive strategies with incomplete information. The proposed architecture consists of three layers: a strategic MARL layer that optimizes high-level defense parameters, a distributed k-winner-take-all coordination layer for low-latency defender selection, and a robust execution layer based on sliding-mode control to preserve physical system stability during reconfiguration. By decoupling strategic adaptation from real-time control, the framework improves scalability and supports resource-aware defense through selective agent activation. Extensive simulations with up to 50 defender agents demonstrate that the proposed approach achieves a defense success rate of 92.4%, reduces the response time by 15% compared with the random MTD, and lowers the energy consumption by 34% on average (up to 52% at N = 50) relative to the flat MARL. These results indicate that hierarchical MARL can significantly enhance CPS resilience by enabling adaptive, efficient, and operationally safe defenses against dynamic cyber-attacks. The proposed framework is particularly suitable for edge-enabled CPS environments with strict, real-time, and safety constraints.