Double Successive Over-Relaxation Q-Learning With an Extension to Deep Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Q-learning (QL) is a fundamental reinforcement learning (RL) algorithm, but its convergence can be slow, particularly with discount factors near one.
• Successive over-relaxation QL (SORQL) accelerates convergence but suffers from model-dependency and overestimation bias in tabular settings.

Purpose of the Study:

• To propose a novel sample-based, model-free double SORQL (MF-DSORQL) algorithm to overcome the limitations of SORQL.
• To theoretically and empirically evaluate the bias and convergence properties of MF-DSORQL.

Main Methods:

• Development of the MF-DSORQL algorithm, a sample-based, model-free approach.
• Theoretical convergence analysis for the tabular setting under boundedness assumptions.
• Extension of MF-DSORQL to large-scale problems using deep reinforcement learning (DRL).

Main Results:

• MF-DSORQL exhibits reduced bias compared to SORQL, both theoretically and empirically.
• Convergence analysis for the tabular MF-DSORQL is provided.
• The DRL extension of MF-DSORQL is validated on benchmark problems.

Conclusions:

• MF-DSORQL offers an effective solution to the limitations of SORQL in reinforcement learning.
• The algorithm is suitable for both tabular and large-scale deep reinforcement learning applications.