A Q-Learning Approach to Flocking With UAVs in a Stochastic Environment

Area of Science:

• Robotics and Autonomous Systems
• Artificial Intelligence
• Control Theory

Background:

• Unmanned aerial vehicles (UAVs) offer efficient solutions for dull, dirty, dangerous, or costly tasks.
• Coordinated multi-UAV systems act as force multipliers, requiring autonomous coordination.
• Swarming behavior in nature provides a model for multi-agent coordination.

Purpose of the Study:

• To investigate model-free reinforcement learning for autonomous flocking in fixed-wing UAVs.
• To develop a control policy enabling flocking in a leader-follower topology.
• To evaluate the learned policies against traditional stochastic optimal control methods.

Main Methods:

• Utilized Peng's Q(λ) algorithm with a variable learning rate for policy learning.
• Modeled the flocking problem as a Markov decision process for small fixed-wing UAVs.
• Introduced stochasticity through environmental disturbances and system uncertainties.

Main Results:

• Demonstrated the feasibility of reinforcement learning for UAV flocking.
• Showcased successful flocking in a leader-follower formation.
• Validated the approach in a non-stationary, stochastic environment.

Conclusions:

• Reinforcement learning effectively enables autonomous flocking behavior in UAVs.
• The leader-follower topology is achievable with learned control policies.
• The proposed method is robust to environmental and system uncertainties.