Minibatch Recursive Least Squares Q-Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Deep Q-network (DQN) is a successful reinforcement learning algorithm but suffers from slow convergence and instability.
• Traditional linear function approximation methods offer faster convergence and stability but struggle with high-dimensional problems.
• Existing DQN improvements rarely leverage traditional methods' strengths.

Purpose of the Study:

• To propose a novel Q-learning algorithm, minibatch recursive least squares Q-learning (MRLS-Q), that integrates the benefits of linear function approximation with DQN.
• To enhance convergence speed and stability in reinforcement learning.
• To create a versatile algorithm applicable to both low- and high-dimensional problems.

Main Methods:

• Developed MRLS-Q, a Q-learning algorithm with linear function approximation.
• Modeled MRLS-Q's learning mechanism and structure similar to DQNs, using states as inputs and incorporating experience replay and minibatch training.
• Implemented an average recursive least squares (RLS) optimization technique for improved convergence.

Main Results:

• MRLS-Q demonstrated effectiveness on the CartPole problem and four Atari games.
• The algorithm showed improved convergence performance, both standalone and when integrated with DQN.
• Experimental analysis investigated the impact of MRLS-Q's hyperparameters.

Conclusions:

• MRLS-Q successfully combines linear function approximation with DQN principles.
• The proposed algorithm offers enhanced convergence and stability for reinforcement learning.
• MRLS-Q is a flexible solution for various problem dimensions and can be seamlessly integrated into existing DQN architectures.