Target Tracking Control of a Biomimetic Underwater Vehicle Through Deep Reinforcement Learning

In this article, the underwater target tracking control problem of a biomimetic underwater vehicle (BUV) is addressed. Since it is difficult to build an effective mathematic model of a BUV due to the uncertainty of hydrodynamics, target tracking control is converted into the Markov decision process and is further achieved via deep reinforcement learning. The system state and reward function of underwater target tracking control are described. Based on the actor-critic reinforcement learning framework, the deep deterministic policy gradient actor-critic algorithm with supervision controller is proposed. The training tricks, including prioritized experience replay, actor network indirect supervision training, target network updating with different periods, and expansion of exploration space by applying random noise, are presented. Indirect supervision training is designed to address the issues of low stability and slow convergence of reinforcement learning in the continuous state and action space. Comparative simulations are performed to show the effectiveness of the training tricks. Finally, the proposed actor-critic reinforcement learning algorithm with supervision controller is applied to the physical BUV. Swimming pool experiments of underwater object tracking of the BUV are conducted in multiple scenarios to verify the effectiveness and robustness of the proposed method.