Stable and Efficient Policy Evaluation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Policy evaluation is crucial for predicting reinforcement learning policy performance.
• Conventional algorithms face challenges with off-policy stability and on-policy efficiency.
• Temporal difference (TD) methods are on-policy efficient but not off-policy stable, while gradient TD and emphatic TD are stable but not efficient.

Purpose of the Study:

• To develop novel policy evaluation algorithms that are both off-policy stable and on-policy efficient.
• To address the limitations of existing temporal difference algorithms in reinforcement learning.

Main Methods:

• Introduction of new algorithms utilizing the oblique projection method.
• Development of techniques to ensure both stability in off-policy settings and efficiency in on-policy scenarios.

Main Results:

• The proposed algorithms demonstrate effectiveness in achieving both off-policy stability and on-policy efficiency.
• Empirical results across various domains validate the superior performance of the novel approach.

Conclusions:

• The novel algorithms effectively overcome the long-standing issues in policy evaluation.
• The oblique projection method offers a promising direction for advancing reinforcement learning algorithms.