Intrinsic Value-Aligned Policy Optimization for Offline-to-Online Reinforcement Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline-to-online reinforcement learning (O2O RL) leverages pretrained policies for efficient adaptation.
• Distribution shift during online finetuning causes significant performance degradation.
• Current methods struggle to balance optimism and pessimism in Q-value estimation.

Purpose of the Study:

• To introduce a novel method, Intrinsic Value-Aligned Policy Optimization (IVPO), to address performance drops in O2O RL.
• To improve Q-value estimation by integrating intrinsic value extraction.
• To enhance policy improvement in O2O RL settings.

Main Methods:

• Developed IVPO, incorporating intrinsic value extraction to compress offline state knowledge.
• Learned an intrinsic value function to guide Q-value updates during online learning.
• Integrated intrinsic and Q-value functions to calibrate estimations and suppress overestimation of out-of-distribution actions.

Main Results:

• IVPO effectively mitigates Q-value estimation errors.
• Achieved state-of-the-art performance on the D4RL benchmark.
• Improved overall task performance by 54.3% across 18 tasks initialized from offline policies.

Conclusions:

• IVPO offers a principled approach to balancing optimism and pessimism in O2O RL.
• The method significantly enhances policy performance by improving Q-value accuracy.
• Theoretical analysis supports IVPO's regret bound and convergence properties.