Action Candidate Driven Clipped Double Q-Learning for Discrete and Continuous Action Tasks.

Area of Science:

• Reinforcement Learning
• Machine Learning
• Artificial Intelligence

Background:

• Double Q-learning is a key algorithm for Markov decision process (MDP) problems.
• Clipped double Q-learning uses a clipped double estimator, but can suffer from underestimation bias in stochastic environments.

Purpose of the Study:

• To propose an action candidate-based clipped double estimator (AC-CDE) to mitigate underestimation bias in Double Q-learning.
• To enhance the performance of Double Q-learning in stochastic environments.

Main Methods:

• Developed an AC-CDE by selecting elite action candidates from one set of estimators and choosing the highest value action from another.
• The AC-CDE clips the action value using the maximum value from the second estimator set.
• Extended the approach to continuous action tasks by approximating elite continuous action candidates.

Main Results:

• Theoretically demonstrated that underestimation bias decreases monotonically with fewer action candidates.
• Showcased that the number of action candidates balances overestimation and underestimation biases.
• Empirically validated improved estimation accuracy on toy environments and strong performance on benchmark problems.

Conclusions:

• The proposed AC-CDE effectively reduces underestimation bias in Double Q-learning.
• The method offers a tunable trade-off between overestimation and underestimation biases.
• The algorithm shows promise for both discrete and continuous action space reinforcement learning tasks.