Progressive learning: A deep learning framework for continual learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Continual learning is crucial for advancing AI, allowing systems to acquire new knowledge without degrading prior learning.
• Catastrophic forgetting and negative forward transfer are significant challenges in continual learning.

Purpose of the Study:

• To introduce and evaluate Progressive Learning, a deep learning framework designed for effective continual learning.
• To demonstrate the framework's ability to manage model capacity and mitigate knowledge interference.

Main Methods:

• Progressive Learning framework employs three procedures: curriculum (task selection), progression (model capacity growth), and pruning (parameter management).
• The progression procedure adds parameters to leverage prior knowledge for new tasks, preventing catastrophic forgetting.
• The pruning procedure counteracts parameter growth and reduces negative forward transfer.

Main Results:

• Progressive Learning was evaluated on image and speech recognition tasks, showing advantages over baseline methods.
• When tasks are related, Progressive Learning achieved faster convergence and superior generalization performance.
• The framework utilized a smaller number of dedicated parameters compared to other methods.

Conclusions:

• Progressive Learning offers an effective approach to continual learning in deep neural networks.
• The framework successfully balances learning new tasks with retaining old knowledge and improving efficiency.
• It demonstrates significant potential for applications requiring continuous adaptation and knowledge integration.