Integrated Double Estimator Architecture for Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Reinforcement learning (RL) algorithms like Q-learning and Deep Q-Networks (DQN) often exhibit estimation bias, specifically overestimation, due to the maximum operation in value estimation.
• Double Q-learning (DQ) and Double DQN mitigate overestimation but can introduce underestimation by employing a double estimator (DE).

Purpose of the Study:

• To propose a novel Integrated Double Estimator (IDE) architecture that balances overestimation and underestimation in RL.
• To introduce two new RL algorithms, Integrated DQ (IDQ) and Integrated DQN (IDDQN), based on the IDE architecture.
• To theoretically analyze estimation bias and prove the unbiasedness of IDE and convergence of IDQ.

Main Methods:

• Proposed an Integrated Double Estimator (IDE) by combining maximum and DE operations for estimating maximum expected action values.
• Developed two RL algorithms: Integrated DQ (IDQ) and its deep network version, Integrated DQN (IDDQN).
• Employed stochastic action selection using one estimator and convex combination of two estimators for action evaluation to eliminate estimation bias.

Main Results:

• Theoretically analyzed the causes of estimation bias in RL and underestimation in DQ.
• Proved the unbiasedness of the proposed IDE and the convergence of the IDQ algorithm.
• Experimental results on grid world and Atari 2600 games demonstrated that IDQ and IDDQN effectively reduce or eliminate estimation bias.

Conclusions:

• IDQ and IDDQN significantly improve learning stability and balance by mitigating estimation bias.
• The proposed IDE architecture offers a promising direction for developing more robust and effective RL algorithms.
• Experiments confirm the practical efficacy of IDQ and IDDQN in enhancing RL performance across various domains.