Reinforcement learning-based formation-surrounding control for multiple quadrotor UAVs pursuit-evasion games

Area of Science:

• Robotics
• Control Systems
• Artificial Intelligence

Background:

• Multiple quadrotor unmanned aerial vehicles (UAVs) are increasingly used in complex scenarios.
• Pursuit-evasion (PE) games present significant control challenges, especially with multiple agents and external disturbances.
• Achieving stable formations and coordinated maneuvers in multi-UAV systems requires advanced control strategies.

Purpose of the Study:

• To develop a reinforcement learning (RL)-based formation-surrounding control method for multiple UAVs in pursuit-evasion (MPE) games.
• To address the challenge of pursuers equally surrounding evaders while maintaining formation integrity under external disturbances.
• To ensure the stability of quadrotor UAVs' position and attitude tracking error subsystems.

Main Methods:

• Proposed a novel control strategy combining feedforward control and reinforcement learning (RL).
• Developed two new cost functions tailored for quadrotor UAVs facing external disturbances.
• Implemented critic-only neural network (NN) weight update laws for optimal cost function estimation.
• Utilized RL to solve Hamilton-Jacobi-Isaacs (HJI) equations for achieving Nash equilibrium.
• Applied Euler's formula to prove the property of equal surrounding.

Main Results:

• Demonstrated the stability of the tracking error subsystem using RL-based control schemes.
• Successfully achieved Nash equilibrium for the multiple quadrotor UAV system.
• Proved the property of equal surrounding for pursuers around evaders for the first time.
• Numerical simulations confirmed the effectiveness and superior performance of the proposed method.

Conclusions:

• The proposed RL-based formation-surrounding control method is effective for multi-UAV pursuit-evasion games.
• The method ensures stable control and achieves the desired equal surrounding formation under external disturbances.
• This research advances multi-agent control systems by integrating RL for complex coordinated behaviors.