A Two-Stage Selective Experience Replay for Double-Actor Deep Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Deep reinforcement learning (DRL) faces challenges in exploration and Q-value estimation accuracy.
• Double-actor architectures offer improvements but suffer from data distribution mismatches during actor updates.
• These mismatches can lead to suboptimal policies in DRL agents.

Purpose of the Study:

• To propose a generic solution to mitigate adverse effects of data distribution differences in double-actor DRL methods.
• To enhance policy learning and overall performance in DRL agents.
• To seamlessly integrate the solution into existing double-actor DRL frameworks.

Main Methods:

• Decomposing double-actor DRL updates into two stages with a consistent sampling approach.
• Utilizing K-means clustering to categorize samples from the replay buffer.
• Employing Jensen-Shannon (JS) divergence to assess distributional differences and prioritize samples for actor updates.

Main Results:

• The proposed method effectively mitigates distribution differences between samples and the current actor being updated.
• Enhanced performance was observed across five state-of-the-art (SOTA) double-actor DRL methods.
• Outperformed eight SOTA single-actor DRL methods on eight benchmark tasks.

Conclusions:

• The novel sampling strategy significantly improves the learning process and policy quality in double-actor DRL.
• This approach offers a robust enhancement for existing DRL frameworks, addressing key exploration and estimation challenges.
• The method demonstrates broad applicability and effectiveness across various DRL applications.