Episodic Memory-Double Actor-Critic Twin Delayed Deep Deterministic Policy Gradient | JoVE Visualize

Area of Science:

Deep Reinforcement Learning
Machine Learning
Robotics and Control Systems

Background:

Deep reinforcement learning (DRL) algorithms often exhibit low sample efficiency, limiting their practical application.
Episodic memory has shown promise in improving sample efficiency for discrete action tasks but faces challenges in high-dimensional continuous action spaces.
Existing methods in continuous control primarily utilize episodic memory information rather than direct action selection guidance.

Purpose of the Study:

To enhance sample efficiency in continuous control tasks by enabling direct use of episodic memory for action selection.
To reduce training steps or increase rewards within a fixed training duration.
To propose a novel framework, Episodic Memory-Double Actor-Critic (EMDAC), for this purpose.

Main Methods:

Proposed the Episodic Memory-Double Actor-Critic (EMDAC) framework, integrating episodic memory for action selection in continuous tasks.
Developed an episodic memory module utilizing a Kalman filter optimizer for weighted experience updates based on time and rewards.
Implemented state-action pair clusters as indices in episodic memory, recording occurrence frequency and value estimates for efficient querying.
Designed an intrinsic reward mechanism based on the novelty of state-action pairs to improve agent exploration.
Applied these modules to the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, resulting in the EMDAC-TD3 algorithm.

Main Results:

EMDAC-TD3 demonstrated significantly higher sample efficiency compared to baseline algorithms in MuJoCo environments.
The algorithm achieved superior final performance over state-of-the-art episodic control and Actor-Critic algorithms.
EMDAC-TD3 showed an average performance improvement of 11.01% over TD3, surpassing current state-of-the-art methods.

Conclusions:

The EMDAC framework effectively leverages episodic memory for action selection in continuous control tasks, enhancing sample efficiency.
EMDAC-TD3 offers a promising approach for improving deep reinforcement learning performance in complex control scenarios.
The proposed methods for episodic memory management and intrinsic reward design contribute to more capable and efficient DRL agents.