Learning without Forgetting.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Training deep learning models like Convolutional Neural Networks (CNNs) often requires large datasets.
• As the number of tasks increases, storing and retraining on all historical data becomes computationally infeasible.
• A key challenge is adding new capabilities to a CNN without access to the original training data for its existing functions.

Purpose of the Study:

• To develop a novel method for incrementally training CNNs on new tasks while retaining previously learned capabilities.
• To address the problem of data scarcity for previously learned tasks when expanding a model's functionality.
• To evaluate the proposed method against existing techniques for model adaptation and knowledge retention.

Main Methods:

• Proposed a Learning without Forgetting (LwF) method that utilizes only new task data for training.
• Focused on preserving the network's original capabilities during the training process for new tasks.
• Compared LwF against feature extraction, fine-tuning, and multitask learning approaches.

Main Results:

• The LwF method demonstrated favorable performance compared to standard feature extraction and fine-tuning techniques.
• LwF achieved performance comparable to multitask learning that assumes access to original task data.
• Surprisingly, LwF showed potential to replace fine-tuning, yielding improved performance on new tasks even with limited old task data.

Conclusions:

• Learning without Forgetting offers an effective solution for continual learning in CNNs when original training data is unavailable.
• The method successfully balances the acquisition of new knowledge with the preservation of existing functionalities.
• LwF presents a promising alternative to traditional fine-tuning, enhancing new task performance while mitigating catastrophic forgetting.