Online selective kernel-based temporal difference learning.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Reinforcement Learning

Background:

• Large-scale and continuous reinforcement learning problems present significant computational challenges.
• Existing methods often struggle with efficiency and convergence in complex environments.
• Value function approximation is crucial for solving these problems.

Purpose of the Study:

• To propose a novel online selective kernel-based temporal difference (OSKTD) learning algorithm.
• To address the computational complexity and efficiency issues in large-scale reinforcement learning.
• To enhance the performance and convergence speed of reinforcement learning algorithms.

Main Methods:

• Developed an online sparsification method based on kernel distance and selective ensemble learning.
• Introduced a selective kernel-based value function that identifies optimal samples.
• Integrated temporal difference (TD) learning with gradient descent for parameter updating.
• Implemented an online sparsification procedure with O(n) complexity.

Main Results:

• OSKTD demonstrated faster convergence to optimal policies in the Maze experiment compared to traditional and state-of-the-art algorithms.
• In the Mountain Car experiment, OSKTD achieved better local optima and faster convergence than traditional methods.
• The algorithm required less computation time than other sparsification methods and reached competitive ultimate optima.

Conclusions:

• The proposed OSKTD algorithm is effective for large-scale and continuous reinforcement learning problems.
• OSKTD offers significant improvements in convergence speed, computational efficiency, and solution quality.
• The novel sparsification and value function selection methods contribute to its enhanced performance.