Efficient offline-to-online reinforcement learning with pre-reduced out-of-distribution Q-values

Offline reinforcement learning (RL) is a data-driven policy learning method, where the results largely depend on the quality of the training dataset. When expert data is hard to collect, it is crucial to improve the performance of the learned policy through online fine-tuning by interacting with the environment. A natural idea is to extend the offline algorithm used in pre-training for fine-tuning, but its conservatism is also inherited, leading to slow improvement. Although using traditional off-policy algorithms for fine-tuning can achieve good final results, it may cause a significant performance drop at the beginning, which is typically due to the overestimation of out-of-distribution (OOD) actions. To make fine-tuning efficient and stable, we propose Pre-Reduced Q-Learning (PRQL). Firstly, PRQL trains a middle policy to seek actions that are overestimated by the offline Q-network. Secondly, a middle Q-network is trained to reduce the value of such actions before fine-tuning. Finally, PRQL fine-tunes using new online data and employs a memory regularization term to prevent forgetting the values learned from offline data. Our experiments on different datasets from various continuous control tasks demonstrate that PRQL outperforms most baselines, achieving the highest performance improvement with relatively small initial policy decline.