Acceleration of reinforcement learning by policy evaluation using nonstationary iterative method

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computational Science

Background:

• Reinforcement learning (RL) typically involves iterative policy evaluation and improvement.
• Existing policy evaluation methods often rely on stationary iterative techniques, limiting learning efficiency.

Purpose of the Study:

• To propose novel algorithms for policy evaluation in reinforcement learning.
• To enhance the learning efficiency of RL algorithms through improved policy evaluation.

Main Methods:

• Development of algorithms based on the Krylov Subspace Method (KSM).
• KSM is a nonstationary iterative method applied to policy evaluation.

Main Results:

• KSM-based algorithms demonstrate tens to hundreds of times greater efficiency than stationary methods.
• The enhanced efficiency of KSM algorithms exceeds prior expectations.

Conclusions:

• Krylov Subspace Method offers a significant advancement for policy evaluation in reinforcement learning.
• Further theoretical and numerical analysis clarifies the efficiency gains of KSM-based approaches.