Mild Policy Evaluation for Offline Actor-Critic.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline actor-critic (AC) algorithms face challenges with distributional shift, leading to overestimated values for out-of-distribution (OOD) actions.
• Existing value-regularized methods mitigate this by learning conservative value functions, often causing performance degradation.

Purpose of the Study:

• To introduce a novel approach, mild policy evaluation (MPE), to address optimistic value estimates in offline AC algorithms.
• To analyze the theoretical properties of MPE, including convergence and value function approximation error.
• To develop and evaluate a mild offline AC (MOAC) algorithm integrating MPE.

Main Methods:

• Proposed MPE by constraining the value difference between target policy actions and offline dataset actions.
• Developed the mild offline AC (MOAC) algorithm by incorporating MPE into off-policy AC.
• Conducted theoretical analysis on MPE's convergence, value function gap, and suboptimality.

Main Results:

• The value function gap in MOAC is bounded by sampling errors.
• Theoretical analysis shows that the true state value function can be recovered without sampling errors.
• Experimental results on the D4RL benchmark dataset confirm MPE's effectiveness and MOAC's superior performance over state-of-the-art offline RL algorithms.

Conclusions:

• Mild policy evaluation (MPE) effectively mitigates optimistic value estimates in offline AC.
• The proposed mild offline AC (MOAC) algorithm demonstrates improved performance and theoretical guarantees.
• MOAC represents a significant advancement in offline reinforcement learning, outperforming existing methods.