A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learning

Area of Science:

• Deep Learning
• Artificial Intelligence
• Machine Learning

Background:

• Forgetting, the loss of acquired knowledge, is prevalent in deep learning beyond continual learning.
• It appears in generative models (generator shifts) and federated learning (heterogeneous data).
• Existing research often views forgetting solely as detrimental.

Purpose of the Study:

• To broaden the understanding of forgetting beyond continual learning.
• To re-evaluate forgetting as a potentially beneficial phenomenon.
• To identify novel strategies for managing forgetting by exploring diverse research fields.

Main Methods:

• Comprehensive literature review across various deep learning domains.
• Analysis of forgetting manifestations in generative and federated learning.
• Comparative study of existing and potential forgetting mitigation/harnessing strategies.

Main Results:

• Forgetting is a multifaceted issue impacting various deep learning applications.
• Forgetting can be advantageous in specific contexts, such as privacy preservation.
• Current approaches to forgetting are often narrowly focused on continual learning.

Conclusions:

• A broader perspective on forgetting is necessary for effective deep learning.
• Forgetting can be a double-edged sword, offering both challenges and opportunities.
• Novel strategies are needed to mitigate, harness, or embrace forgetting in real-world applications.