Memory Recall: A Simple Neural Network Training Framework Against Catastrophic Forgetting.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neuroscience

Background:

• Biological intelligence exhibits continuous learning without forgetting.
• Deep neural networks (NNs) often suffer from catastrophic forgetting, losing prior knowledge when learning new tasks.
• Continual learning aims to train NNs without this detrimental forgetting.

Purpose of the Study:

• To propose a novel and simple framework, Memory Recall (MeRec), for continual learning in deep NNs.
• To address the catastrophic forgetting problem in neural networks.
• To develop a method inspired by brain mechanisms like memory replay and synaptic consolidation.

Main Methods:

• Analyzing feature stability across tasks in NNs to identify task-stable features.
• Utilizing a memory module to store feature statistics (mean and standard deviation) for each learned task.
• Implementing a replay strategy with Gaussian distribution-based feature regeneration to recall previous task knowledge.
• Incorporating weight regularization to preserve previously learned weights.

Main Results:

• MeRec demonstrated leading performance in continual learning on CIFAR-10 and CIFAR-100 datasets.
• Achieved significant reduction in accuracy drop (at least 50%) after multiple tasks compared to state-of-the-art methods.
• Required an extremely small memory budget, storing only two feature vectors per class.

Conclusions:

• The proposed MeRec framework effectively mitigates catastrophic forgetting in deep neural networks.
• MeRec offers a computationally efficient and memory-light solution for continual learning.
• The findings suggest that leveraging feature stability and memory replay is a promising direction for robust continual learning systems.