Synaptic metaplasticity in binarized neural networks.

Area of Science:

• Computational Neuroscience
• Deep Learning
• Artificial Intelligence

Background:

• Deep neural networks excel but suffer from catastrophic forgetting, rapidly losing prior knowledge when learning new tasks.
• Neuroscience suggests synaptic metaplasticity (adjusting plasticity based on history) mitigates forgetting in the brain, but direct application to deep learning is challenging.

Purpose of the Study:

• To develop a novel training technique for deep neural networks that mitigates catastrophic forgetting.
• To bridge computational neuroscience insights on metaplasticity with deep learning architectures, specifically binarized neural networks.

Main Methods:

• Interpreting hidden weights in binarized neural networks as metaplastic variables.
• Modifying training techniques to incorporate these metaplastic variables.
• Experimental validation in multitask and stream learning scenarios.
• Theoretical analysis on a tractable task.

Main Results:

• Demonstrated a training technique reducing catastrophic forgetting in binarized neural networks.
• Achieved performance comparable to mainstream techniques, even without explicit task boundaries or prior data.
• Showcased effectiveness in both multitask and stream learning settings.

Conclusions:

• The proposed metaplasticity-inspired training effectively alleviates catastrophic forgetting in deep learning.
• This approach offers a promising direction for developing more robust AI systems, particularly for embedded and neuromorphic applications.
• The work highlights the potential of novel nanodevices with metaplasticity-analogous physics for future AI hardware.