Eliminating Primacy Bias in Online Reinforcement Learning by Self-Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning

Background:

• Deep reinforcement learning is prone to overfitting early in training.
• This overfitting, known as primacy bias, leads to suboptimal agent decisions and exploration.
• Primacy bias obstructs agents from learning generalized policies for future states.

Purpose of the Study:

• To systematically investigate the causes and characteristics of primacy bias in online reinforcement learning.
• To develop a novel framework, self-distillation reinforcement learning (SDRL), to overcome primacy bias.
• To enhance policy generalization and accelerate learning in reinforcement learning agents.

Main Methods:

• Developed a self-distillation reinforcement learning (SDRL) framework based on knowledge distillation.
• Agents transfer learned knowledge to a new, randomly initialized policy at regular intervals.
• Incorporated L2 regularization and a self-imitation mechanism to prevent overfitting and accelerate learning.

Main Results:

• Experimental results in DMC and Atari 100k demonstrate SDRL's effectiveness in eliminating primacy bias.
• The proposed method enables the generation of more generalized policies.
• Policies refined through knowledge distillation lead to faster score improvements for agents.

Conclusions:

• Self-distillation reinforcement learning effectively addresses primacy bias in online reinforcement learning.
• SDRL promotes the development of robust and generalized policies.
• The framework enhances learning efficiency and agent performance in complex environments.