Dual experience replay enhanced deep deterministic policy gradient for efficient continuous data sampling.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Distributed reinforcement learning (RL) systems face challenges with sample utilization and policy stability due to network delays and asynchronous updates.
• Existing methods struggle with robust learning under asynchronous conditions, limiting scalability in real-world applications.

Purpose of the Study:

• To propose TPDEB, a dual experience replay framework designed to improve sample efficiency and policy stability in asynchronous distributed RL.
• To enhance convergence speed and robustness in continuous control tasks.

Main Methods:

• TPDEB utilizes a dual-buffer strategy combining standard and prioritized experience replay buffers.
• It incorporates trajectory-level prioritized sampling for high-value experiences and KL-regularized learning to constrain policy drift.
• The framework scales parallel actor replicas to collect diverse and redundant experience.

Main Results:

• TPDEB demonstrated superior convergence speed and final performance compared to baseline distributed RL algorithms on MuJoCo benchmarks.
• Performance gains were particularly notable under constrained actor-learner bandwidth conditions.
• Ablation studies confirmed the effectiveness of trajectory-level prioritization and KL-regularization in improving learning.

Conclusions:

• TPDEB offers a practical and scalable solution for asynchronous distributed reinforcement learning.
• The dual-buffer approach and specific regularization techniques effectively address limitations in sample utilization and policy instability.
• This framework significantly advances the robustness and efficiency of RL in complex control tasks.