Asynchronous Episodic Deep Deterministic Policy Gradient: Toward Continuous Control in Computationally Complex...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Robotics

Background:

• Deep Deterministic Policy Gradient (DDPG) is effective for continuous control but struggles with data and training inefficiency in complex environments.
• Increasing parallelism in asynchronous reinforcement learning (RL) can diminish sample efficiency and training stability.

Purpose of the Study:

• To introduce Asynchronous Episodic Deep Deterministic Policy Gradient (AE-DDPG) to address DDPG's limitations.
• To enhance learning effectiveness and reduce training time in computationally complex RL tasks.

Main Methods:

• Implemented an asynchronous data collection scheme.
• Redesigned experience replay with episodic control for rapid learning from good trajectories.
• Introduced novel action space noise for improved exploration.

Main Results:

• AE-DDPG achieved higher rewards and lower time consumption in a computationally complex learning-to-run task.
• Demonstrated 2x-4x improvement in sample efficiency in MuJoCo environments compared to DDPG variants.
• Ablation studies confirmed the effectiveness of individual AE-DDPG components.

Conclusions:

• AE-DDPG offers a more efficient and effective approach to continuous control tasks in challenging environments.
• The proposed methods for asynchronous data collection, episodic replay, and exploration noise significantly improve RL performance.