Mitigating OOD overoptimism via in-sample value function in offline reinforcement learning.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Offline Reinforcement Learning (RL) learns from pre-recorded data without online interaction.
• Evaluating out-of-distribution (OOD) actions in offline RL is difficult, often causing over-optimistic value estimations.
• Existing in-sample learning methods avoid OOD actions but limit generalization capacity.

Purpose of the Study:

• To develop a method that accurately evaluates OOD actions in offline RL.
• To improve the generalization capacity of in-sample learning methods.
• To introduce a novel regularization technique for offline RL.

Main Methods:

• Proposed In-sample Expectile Value Regularization (IEVR) method.
• Utilized in-sample expectile values to constrain OOD action-values.
• Preserved standard Bellman updates for in-sample actions.
• Provided theoretical analysis of IEVR convergence.

Main Results:

• IEVR effectively constrains OOD actions using in-sample expectile values.
• Theoretical analysis confirmed IEVR's convergence properties.
• Experimental results demonstrated significant performance improvements over existing methods.
• IEVR showed effectiveness across diverse tasks in the D4RL benchmark.

Conclusions:

• IEVR offers a concise and effective solution for OOD action evaluation in offline RL.
• In-sample expectile values serve as an effective regularization method.
• The proposed method bridges the gap between safe in-sample learning and generalization.