Subspace distillation for continual learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Continual learning aims to retain knowledge from previous tasks when learning new ones.
• Catastrophic forgetting is a major challenge, where neural networks lose previously acquired information.

Purpose of the Study:

• To propose a novel knowledge distillation technique for continual learning.
• To mitigate catastrophic forgetting by leveraging the manifold structure of neural network spaces.

Main Methods:

• Approximating the data manifold using first-order linear subspaces.
• Employing knowledge distillation informed by manifold structure.
• Applying the method to both classification and segmentation tasks.

Main Results:

• The subspace modeling approach demonstrates robustness to noise, effectively mitigating catastrophic forgetting.
• The proposed method outperforms existing continual learning techniques on datasets like Pascal VOC and Tiny-Imagenet.
• The method can be integrated with other learning approaches to enhance performance.

Conclusions:

• The novel knowledge distillation technique effectively addresses catastrophic forgetting in continual learning.
• Leveraging manifold structure offers a promising direction for robust continual learning.
• The proposed method shows broad applicability and compatibility with existing approaches.