Kernel-Based Least Squares Temporal Difference With Gradient Correction.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Policy evaluation is crucial for reinforcement learning (RL).
• Existing algorithms like Temporal Difference (TD) methods face challenges in convergence and parameter tuning.
• Need for robust and efficient policy evaluation algorithms in RL.

Purpose of the Study:

• Introduce novel policy evaluation algorithms: Least Squares Temporal Difference with Gradient Correction (LS-TDC) and its kernel-based version (KLS-TDC).
• Enhance convergence performance and robustness compared to existing methods.
• Develop an efficient approach for feature selection and control learning problems.

Main Methods:

• LS-TDC algorithm derived from TDC, minimizing mean-square projected Bellman error.
• Kernel-based LS-TDC (KLS-TDC) utilizing kernel methods for automatic feature vector selection.
• Approximate linear dependence analysis for kernel sparsification.
• Policy iteration strategy motivated by KLS-TDC for control learning.

Main Results:

• LS-TDC and KLS-TDC demonstrate superior approximation and convergence performance over existing RL algorithms.
• Both algorithms exhibit higher efficiency in sample usage.
• Reduced sensitivity to parameter tuning and a smaller tuning burden were observed.
• KLS-TDC facilitates automatic feature selection through kernel methods.

Conclusions:

• LS-TDC and KLS-TDC are effective policy evaluation algorithms for reinforcement learning.
• These methods offer significant improvements in performance, efficiency, and robustness.
• KLS-TDC provides an automated feature selection mechanism, simplifying complex RL tasks.