Continual Learning: Forget-free Winning Subnetworks for Video Representations.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• The Lottery Ticket Hypothesis (LTH) suggests dense neural networks contain smaller, efficient subnetworks.
• Continual learning aims to update models with new data without forgetting previous knowledge.
• Existing methods face challenges in efficient weight reuse and preventing overfitting in incremental learning scenarios.

Purpose of the Study:

• To investigate the effectiveness of Winning Subnetworks (WSNs) for various continual learning tasks.
• To adapt WSNs for scenarios like Few-Shot Class Incremental Learning (FSCIL) and Video Incremental Learning (VIL).
• To integrate Fourier Subneural Operators (FSO) within WSNs for enhanced feature encoding and subnetwork reuse.

Main Methods:

• Leveraging pre-existing weights from dense networks to form WSNs for Task Incremental Learning (TIL) and Task-agnostic Incremental Learning (TaIL).
• Introducing Soft subnetworks (SoftNets) as a variation of WSNs to mitigate overfitting in FSCIL.
• Incorporating Fourier Subneural Operators (FSO) for compact video encoding and identifying reusable subnetworks in Video Incremental Learning (VIL).
• Applying FSO within WSN frameworks across VIL, TIL, and FSCIL.

Main Results:

• WSNs demonstrate efficient learning by reusing weights from dense networks.
• SoftNets effectively prevent overfitting in data-scarce FSCIL settings.
• FSO integration significantly improves task performance in continual learning.
• FSO enhances higher-layer representations in TIL and FSCIL, and lower-layer representations in VIL.

Conclusions:

• WSNs offer an efficient approach to continual learning, inspired by the LTH.
• FSO is a valuable component for enhancing WSNs, particularly in video and few-shot learning.
• The proposed methods show significant improvements in task performance across diverse continual learning benchmarks.