Autonomous maneuver decision-making method based on reinforcement learning and Monte Carlo tree search

Area of Science:

• Artificial Intelligence
• Robotics
• Aerospace Engineering

Background:

• Autonomous maneuver decision-making in air combat traditionally relies on human expertise, limiting AI capabilities.
• Existing methods using advantage or objective functions hinder progress in unmanned combat aerial vehicle (UCAV) decision-making.

Purpose of the Study:

• To investigate a maneuver decision-making method for UCAVs that does not require human knowledge or predefined advantage functions.
• To develop and validate a novel approach combining deep reinforcement learning and Monte Carlo tree search for autonomous air combat.

Main Methods:

• Proposed a maneuver decision-making method integrating deep reinforcement learning with Monte Carlo tree search (MCTS) in a continuous action space.
• Utilized neural network-guided MCTS with self-play to train UCAV agents, starting from random behaviors.
• Generated combat data through self-play to train neural networks, iteratively selecting higher-performing networks via simulation.

Main Results:

• The proposed method demonstrated effectiveness in simulations, even when using a detailed missile kinematic model.
• Simulations with both fixed and random initial states confirmed the method's efficiency.
• The approach meets real-time requirements for autonomous air combat maneuver decision-making.

Conclusions:

• Autonomous maneuver decision-making for air combat can be achieved without relying on human knowledge or advantage functions.
• The integration of deep reinforcement learning and Monte Carlo tree search provides an efficient and effective solution for UCAV decision-making.
• The developed method shows significant potential for advancing autonomous capabilities in air combat.