Offline Reward Perturbation Boosts Distributional Shift in Online RL

Area of Science:

• Artificial Intelligence
• Machine Learning
• Reinforcement Learning

Background:

• Offline-to-online reinforcement learning (RL) reduces online sample complexity by leveraging pre-collected offline data.
• This approach introduces new vulnerabilities to data poisoning attacks targeting the offline training phase.

Purpose of the Study:

• To reveal vulnerabilities in critic-regularized offline RL against novel data poisoning attacks.
• To propose a stealthy attack method that maintains performance during offline training but degrades online fine-tuning.

Main Methods:

• Leveraging bi-level optimization techniques.
• Promoting over-estimation and distribution shift within the offline-to-online RL framework.
• Developing a stealthy attack with a small budget and without white-box access.

Main Results:

• Demonstrated stealthiness: offline training performance remains intact.
• Significant performance degradation observed during the online fine-tuning stage.
• Attack effectiveness confirmed across four experimental environments.

Conclusions:

• Critic-regularized offline-to-online RL is vulnerable to stealthy data poisoning attacks.
• The proposed bi-level optimization method effectively exploits these vulnerabilities.
• The attack is practical, requiring minimal resources and no white-box access.