A clustering-based graph Laplacian framework for value function approximation in reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Control Theory

Background:

• Sequential decision problems with large or continuous state spaces are challenging in reinforcement learning (RL).
• Feature representation and value function approximation (VFA) are critical research areas for addressing these challenges.
• Existing methods may require extensive data or computational resources.

Purpose of the Study:

• To present a clustering-based graph Laplacian framework for feature representation and VFA in RL.
• To enable efficient handling of continuous state spaces in Markov decision processes (MDPs).
• To improve the performance and sample efficiency of RL algorithms.

Main Methods:

• Constructing a graph Laplacian using clustering techniques (K-means, Fuzzy C-means) via subsampling in continuous state MDPs.
• Generating basis functions for VFA through spectral analysis of the graph Laplacian.
• Integrating the framework with representation policy iteration (RPI) algorithms.

Main Results:

• The proposed approach automatically generates basis functions for VFA.
• Fewer sample points are needed to compute efficient basis functions compared to previous RPI methods.
• Improved learning control performance was observed across various parameter settings.

Conclusions:

• The clustering-based graph Laplacian framework offers an efficient solution for feature representation and VFA in RL with continuous state spaces.
• This method enhances sample efficiency and control performance.
• The framework provides a robust approach for tackling complex sequential decision problems.