Kernel Recursive Least-Squares Temporal Difference Algorithms with Sparsification and Regularization.

Area of Science:

• Machine Learning
• Reinforcement Learning
• Kernel Methods

Background:

• Least-squares temporal difference (LSTD) algorithms with sparse kernel methods offer automatic feature dictionary construction and improved generalization.
• Existing kernel-based LSTD methods lack regularization and employ batch/offline sparsification, limiting their use in online learning.

Purpose of the Study:

• To propose two novel kernel recursive LSTD algorithms designed for online learning scenarios.
• To address limitations of previous methods by incorporating regularization and efficient online processing techniques.

Main Methods:

• Integration of online sparsification for unknown state regions.
• Inclusion of L2 and L1 regularization to prevent overfitting and noise.
• Application of recursive least squares to reduce computational complexity.
• Utilization of a sliding-window approach to manage computational cost.
• Implementation of fixed-point subiteration and online pruning for L1 regularization.

Main Results:

• The developed algorithms demonstrate effectiveness in simulation experiments.
• The proposed methods successfully handle unknown state regions and noise.
• Computational complexity and cost are significantly reduced compared to traditional approaches.

Conclusions:

• The novel kernel recursive LSTD algorithms offer a robust and efficient solution for online learning problems.
• The combination of techniques like online sparsification and regularization enhances performance and applicability.
• The algorithms show promise for real-world applications requiring adaptive learning.