Kernel-Based Decentralized Policy Evaluation for Reinforcement Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Optimization Theory

Background:

• Decentralized learning is crucial for multi-agent reinforcement learning (RL).
• Nonparametric methods offer flexibility but pose computational challenges.
• Policy evaluation requires accurate state-value function estimation.

Purpose of the Study:

• To develop a decentralized nonparametric method for policy evaluation in RL.
• To analyze the statistical convergence properties of the proposed method.
• To address computational and communication feasibility in multi-agent settings.

Main Methods:

• Utilizing a regression-based multistage iteration technique.
• Employing infinite-dimensional gradient descent (GD) in a reproducing kernel Hilbert space (RKHS).
• Applying Nyström approximation for finite-dimensional projection to enhance feasibility.

Main Results:

• Establishing the first statistical error bounds for value function estimation in a fully decentralized nonparametric framework.
• Demonstrating the convergence of the proposed method.
• Comparing the regression-based approach with the kernel temporal difference (TD) method through numerical studies.

Conclusions:

• The proposed method provides a statistically sound and computationally feasible solution for decentralized nonparametric policy evaluation.
• The established error bounds offer theoretical guarantees for the convergence of value function estimation.
• This work advances the understanding and application of RL in complex multi-agent systems.