HyperMask: Adaptive hypernetwork-based masks for continual learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Artificial neural networks face catastrophic forgetting when trained sequentially on multiple tasks.
• Hypernetwork-based approaches are effective continual learning (CL) strategies but can generate disparate architectures.
• The lottery ticket hypothesis suggests sparse subnetworks ('winning tickets') can maintain network performance.

Purpose of the Study:

• To introduce HyperMask, a method addressing hypernetwork limitations in continual learning.
• To leverage the lottery ticket hypothesis for dynamic network filtering in CL.
• To enable a single network with weighted subnets for enhanced task-specific performance.

Main Methods:

• HyperMask utilizes a hypernetwork to generate semi-binary masks for dynamic target network filtering.
• The method dynamically enhances or diminishes weight significance based on the CL task.
• It applies the lottery ticket hypothesis to create dedicated subnetworks within a single network.

Main Results:

• HyperMask achieves competitive results across various continual learning datasets.
• The proposed method demonstrates state-of-the-art performance in specific CL scenarios.
• Effectiveness is shown for both derived and unknown task identities.

Conclusions:

• HyperMask offers an effective solution to catastrophic forgetting in continual learning.
• The dynamic filtering approach, inspired by the lottery ticket hypothesis, enhances CL performance.
• HyperMask provides a robust and adaptable strategy for sequential task learning.