A unified framework to control estimation error in reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Accurate Q-value estimation is vital for optimal policy acquisition in reinforcement learning.
• Existing Actor-Critic methods often exhibit underestimation bias, impacting performance.
• Critic initialization introduces significant estimation bias irrespective of the method used.

Purpose of the Study:

• To propose CEILING, a novel framework to mitigate Q-value estimation errors in model-free Actor-Critic methods.
• To enhance the accuracy and stability of Q-value estimation by addressing underestimation bias.
• To introduce two implementations: Direct Picking Operation and Exponential Softmax Weighting Operation.

Main Methods:

• CEILING incorporates true Q-values (via Monte Carlo) during training to evaluate estimation methods.
• Direct Picking Operation selects the best method at fixed intervals.
• Exponential Softmax Weighting Operation dynamically weights methods based on accuracy.

Main Results:

• Theoretical analysis demonstrates more accurate and stable Q-value estimation with CEILING.
• The upper bound of estimation bias is analyzed.
• Proposed algorithms achieve superior performance on benchmark tasks.

Conclusions:

• CEILING offers a simple, compatible framework to improve Q-value estimation in Actor-Critic methods.
• The proposed methods effectively reduce underestimation bias and enhance learning stability.
• CEILING demonstrates significant performance improvements in benchmark reinforcement learning tasks.