Reducing Estimation Bias via Triplet-Average Deep Deterministic Policy Gradient.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Reinforcement Learning

Background:

• Q-learning algorithms often suffer from overestimation bias in single-critic models, impacting performance.
• Underestimation bias, prevalent in double-critic Q-learning, has been less explored but also affects performance.

Purpose of the Study:

• Investigate the underestimation phenomenon in the twin delayed deep deterministic actor-critic (TD3) algorithm.
• Propose a novel algorithm to mitigate estimation bias in deep deterministic policy gradient (DDPG) methods.

Main Methods:

• Theoretical demonstration of underestimation bias in TD3.
• Development of a triplet-average DDPG algorithm using weighted action values from three target critics.
• Averaging previous target values to reduce per-update approximation error.

Main Results:

• Empirical evidence confirms that underestimation bias negatively impacts TD3 performance.
• The proposed triplet-average DDPG algorithm demonstrates superior performance across various continuous control tasks.
• The method effectively reduces estimation bias and approximation error.

Conclusions:

• Underestimation bias is a significant issue in double-critic Q-learning, affecting algorithms like TD3.
• The triplet-average DDPG algorithm offers an effective solution for bias reduction in deep reinforcement learning.
• This approach advances the state-of-the-art in continuous control tasks.