Variational Data-Free Knowledge Distillation for Continual Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks face catastrophic forgetting in continual learning.
• Existing methods often require storing previous task data, posing privacy and security risks.
• Realistic continual learning scenarios limit access to past data and memory resources.

Purpose of the Study:

• To develop a privacy-preserving continual learning method without storing previous task data.
• To mitigate catastrophic forgetting under limited memory and data constraints.
• To enhance sequential task learning performance in deep neural networks.

Main Methods:

• A novel knowledge distillation approach within an information-theoretic framework.
• Maximizing the variational lower bound of mutual information using graph convolutional networks.
• Employing Taylor expansion for a novel regularizer based on compressed gradients.
• Integrating self-supervised learning for effective feature extraction.

Main Results:

• Achieved state-of-the-art performance on continual learning benchmarks.
• Demonstrated effectiveness in image classification and semantic segmentation tasks.
• Successfully avoided catastrophic forgetting without storing previous task data or networks.

Conclusions:

• The proposed method offers an effective solution for privacy-preserving continual learning.
• It addresses the challenges of catastrophic forgetting in realistic, resource-constrained environments.
• The approach advances the field of continual learning by enabling efficient sequential task acquisition.