Catastrophic Interference in Reinforcement Learning: A Solution Based on Context Division and Knowledge Distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) power reinforcement learning (RL) agents for complex control tasks.
• Standard DNNs assume independent and identically distributed (i.i.d.) inputs, which is violated in RL due to temporal correlations and nonstationarity.
• This violation can cause catastrophic interference and performance degradation in RL agents.

Purpose of the Study:

• To mitigate catastrophic interference in single-task deep reinforcement learning.
• To enhance the stability and performance of deep Q-learning agents in nonstationary environments.

Main Methods:

• Introduced interference-aware deep Q-learning (IQ), a novel approach for deep RL.
• Employed online clustering for dynamic context division during training.
• Utilized a multihead network architecture and knowledge distillation regularization to preserve context-specific policies.

Main Results:

• IQ demonstrated consistent improvements in stability and performance compared to existing methods.
• The proposed method was validated through extensive experiments on classic control and Atari tasks.
• IQ effectively addresses catastrophic interference in deep Q-learning.

Conclusions:

• Interference-aware deep Q-learning (IQ) offers a robust solution to catastrophic interference in single-task deep RL.
• The method enhances agent performance and stability in environments with nonstationary data distributions.
• IQ provides a valuable advancement for developing more reliable deep RL systems.