Diversity Evolutionary Policy Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Policy gradient reinforcement learning algorithms risk local optima due to gradient disappearance, hindering agent exploration.
• Existing methods struggle with maintaining policy diversity, limiting performance in complex continuous control tasks.

Purpose of the Study:

• To propose a novel algorithm, Diversity Evolutionary Policy Deep Reinforcement Learning (DEPRL), to address the local optima problem in reinforcement learning.
• To enhance the exploration capabilities of reinforcement learning agents by promoting policy diversity.

Main Methods:

• Combined Cross-Entropy Method (CEM), Maximum Mean Discrepancy (MMD), and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm.
• Utilized MMD to measure policy distance, encouraging maximization of return and inter-policy distance during gradient updates.
• Incorporated cumulative returns and policy distance into population fitness to promote offspring diversity.

Main Results:

• DEPRL demonstrated excellent performance on continuous control tasks within the MuJoCo environment.
• Achieved a significant performance improvement, nearly 20% increase in return, compared to TD3 in the Ant-v2 environment.
• Effectively reduced the risk of falling into local optima by enhancing policy exploration.

Conclusions:

• DEPRL successfully mitigates the local optima issue in reinforcement learning through enhanced policy diversity.
• The proposed method offers a promising approach for improving the performance and exploration capabilities of deep reinforcement learning agents in continuous control.