Multi Pseudo Q-Learning-Based Deterministic Policy Gradient for Tracking Control of Autonomous Underwater Vehicles

Area of Science:

• Robotics
• Control Systems
• Artificial Intelligence

Background:

• Underactuated autonomous underwater vehicles (AUVs) face challenges in trajectory tracking due to unknown dynamics and input constraints.
• Existing policy gradient methods with single actor-critic architectures often struggle with satisfactory tracking accuracy and stable learning.

Purpose of the Study:

• To develop an advanced algorithm for high-level trajectory tracking control accuracy and stable learning in underactuated AUVs.
• To address limitations of current methods by proposing a hybrid actors-critics architecture.

Main Methods:

• A hybrid actors-critics architecture is employed, training multiple actors and critics for deterministic policy and action-value function learning.
• Critics utilize an expected absolute Bellman error-based updating rule to select the worst critic for updates.
• Pseudo Q-learning and Multi Pseudo Q-learning (MPQ) are developed for continuous action spaces to reduce action-value function overestimation and stabilize learning.
• Deterministic policy gradient is applied to actors, with the final policy being an average of all actors to prevent drastic updates.

Main Results:

• The proposed MPQ-based deterministic policy gradient (MPQ-DPG) algorithm demonstrates effectiveness and generality in AUV trajectory tracking.
• The algorithm achieves high-level tracking control accuracy and stable learning across different reference trajectories.
• Increasing the number of actors and critics further enhances the performance of the MPQ-DPG algorithm.

Conclusions:

• The MPQ-DPG algorithm offers a significant improvement for trajectory tracking in underactuated AUVs with unknown dynamics.
• The hybrid actors-critics approach, combined with MPQ, effectively enhances control accuracy and learning stability.
• The findings suggest that the proposed method provides a robust solution for complex AUV control tasks.