Hierarchical Reinforcement Learning for UAV-PE Game With Alternative Delay Update Method.

Area of Science:

• Robotics and Control Systems
• Artificial Intelligence
• Game Theory

Background:

• Unmanned Aerial Vehicle (UAV) pursuit-evasion (PE) games present complex control challenges.
• Existing methods often struggle with the coupled kinematics and dynamics inherent in these systems.
• Efficiently computing Nash equilibrium (NE) solutions is crucial for developing robust UAV-PE strategies.

Purpose of the Study:

• To propose a novel hierarchical reinforcement learning (HRL) algorithm for UAV-PE game systems.
• To integrate an alternative delay update (ADU) method for enhanced training efficiency.
• To obtain approximate Nash equilibrium (NE) solutions for UAV-PE games with coupled kinematics and dynamics.

Main Methods:

• A hierarchical learning process involving zero-sum kinematics and optimal dynamics games.
• Deep neural networks (NNs) to approximate policy and value functions at kinematic and dynamic levels.
• The ADU method to stabilize training by fixing one player's strategy.

Main Results:

• Development of an HRL algorithm with ADU for UAV-PE systems.
• Derivation of sufficient conditions for algorithm convergence and optimality.
• Obtained overload inequalities ensuring dynamic state tracking via kinematic control input.

Conclusions:

• The proposed HRL algorithm with ADU is feasible and effective for UAV-PE games.
• The method successfully computes approximate NE solutions for coupled systems.
• Simulation results validate the algorithm's performance and training efficiency.