Prioritized experience replay based on dynamics priority.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Experience replay significantly advances reinforcement learning by improving data utilization.
• Prioritized Experience Replay (PER) enhances sampling efficiency by prioritizing experiences with high temporal difference (TD) errors.
• Existing PER methods often use fixed or linear combinations of priority criteria, neglecting dynamic value changes during training.

Purpose of the Study:

• To introduce a novel prioritized experience replay algorithm, PERDP, with a dynamic priority adjustment framework.
• To address the limitations of static priority criteria in existing PER algorithms.
• To improve the convergence speed of reinforcement learning agents.

Main Methods:

• Developed PERDP, a novel algorithm that adaptively adjusts the weights of priority criteria.
• PERDP evaluates experience value based on the current network and the average priority level of the experience pool.
• Implemented and tested PERDP within the Soft Actor-Critic (SAC) model in the OpenAI Gym environment.

Main Results:

• PERDP demonstrated a superior convergence speed compared to standard PER.
• The dynamic adjustment framework of PERDP effectively handles the changing value of experiences during training.
• Experimental results validated the enhanced performance of PERDP in the tested reinforcement learning tasks.

Conclusions:

• PERDP offers a more effective approach to prioritized experience replay by incorporating dynamic priority adjustments.
• The proposed method enhances learning efficiency and convergence speed in reinforcement learning.
• PERDP represents a significant advancement over traditional PER algorithms by accounting for the evolving nature of experience value.