Entropy-Regularized Hierarchical MARL for Resilient Moving Target Defense in Cyber-Physical Systems

Area of Science:

• Cybersecurity
• Artificial Intelligence
• Control Systems Engineering

Background:

• Cyber-Physical Systems (CPS) face evolving cyber threats, necessitating secure and stable operations.
• Current moving target defense (MTD) methods lack scalability and real-time operational resilience.
• Existing MTD architectures struggle with adaptive strategies and incomplete information handling.

Purpose of the Study:

• To propose a resilience-oriented hierarchical multi-agent reinforcement learning (MARL) framework for adaptive MTD in CPS.
• To enhance the scalability and efficiency of MTD in dynamic cyber-attack scenarios.
• To ensure operational safety and stability during system reconfigurations under real-time constraints.

Main Methods:

• Modeling attacker-defender dynamics as a partially observable stochastic game.
• Implementing a three-layer architecture: strategic MARL, k-winner-take-all coordination, and sliding-mode control execution.
• Decoupling strategic adaptation from real-time control for improved scalability and resource-aware defense.

Main Results:

• Achieved a 92.4% defense success rate in simulations with up to 50 defender agents.
• Reduced response time by 15% compared to random MTD and lowered energy consumption by 34% on average versus flat MARL.
• Demonstrated significant improvements in defense efficiency and energy savings with increasing agent numbers.

Conclusions:

• Hierarchical MARL significantly enhances CPS resilience through adaptive, efficient, and safe defenses against dynamic cyber-attacks.
• The proposed framework is well-suited for edge-enabled CPS environments with strict real-time and safety requirements.
• Decoupled architecture enables scalable and resource-aware MTD strategies.