An Improved Distributed Sampling PPO Algorithm Based on Beta Policy for Continuous Global Path Planning Scheme.

Area of Science:

• Maritime Navigation and Robotics
• Artificial Intelligence and Machine Learning

Background:

• Traditional path planning methods in discrete action spaces lead to incomplete ship navigation strategies.
• Reinforcement learning (RL) for path planning suffers from low success rates due to unbalanced sample collection and suboptimal reward functions.

Purpose of the Study:

• To develop an improved path planning framework for ship navigation that addresses limitations of existing methods.
• To enhance the robustness, success rate, and efficiency of RL-based ship path planning strategies.

Main Methods:

• Designed a Box2D physics engine environment with a reward function prioritizing distance to the arrival point and incorporating a potential field.
• Utilized Proximal Policy Optimization (PPO) as a baseline for global path planning.
• Proposed a Beta policy-based distributed sample collection PPO algorithm, dividing regions for distributed sample collection to overcome unbalanced data.

Main Results:

• The distributed sample collection training policy demonstrated superior robustness within the PPO algorithm.
• The Beta policy for action sampling achieved higher path planning success rates and reward accumulation than Gaussian policies.
• The proposed algorithm generated smoother paths compared to traditional algorithms like A*, IDA*, and Dijkstra for equivalent path lengths.

Conclusions:

• The Beta policy-based distributed sample collection PPO algorithm offers a significant advancement in ship navigation path planning.
• This novel approach effectively addresses issues of incomplete strategies, unbalanced samples, and suboptimal rewards in RL-based navigation.