Overcoming Long-Term Catastrophic Forgetting Through Adversarial Neural Pruning and Synaptic Consolidation

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neuroscience

Background:

• Artificial neural networks (ANNs) face catastrophic forgetting, where learning new tasks degrades performance on previously learned ones.
• This forgetting issue intensifies with longer task sequences (long-term catastrophic forgetting).
• Existing models struggle as the shared parameter space shrinks and cumulative error increases with more tasks.

Purpose of the Study:

• To address long-term catastrophic forgetting in ANNs.
• To develop a novel mechanism inspired by mammalian brain memory consolidation.
• To enhance the efficiency and applicability of ANNs in sequential learning scenarios.

Main Methods:

• Propose Adversarial Neural Pruning and synaptic Consolidation (ANPyC), a confrontation mechanism.
• Neural pruning acts as long-term depression, removing task-irrelevant parameters.
• Synaptic consolidation acts as long-term potentiation, strengthening task-relevant parameters via structure-aware importance measurement and element-wise updates.

Main Results:

• ANPyC balances parameter pruning and strengthening, retaining crucial parameters and freeing others for new tasks.
• The method effectively prevents forgetting important information and enables efficient learning of numerous tasks.
• Demonstrated effectiveness and generalization across image classification and generation tasks using various network architectures (MLP, CNN, GAN, VAE).

Conclusions:

• ANPyC overcomes long-term catastrophic forgetting by promoting sparse and polarized synaptic structures.
• The approach enhances long-term learning and memory capabilities in neural networks.
• The proposed method offers a robust solution for sequential task learning in ANNs.