Continual learning with Bayesian compression for shared and private latent representations.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Continual learning methods like Shared and Private Latent Representations (SPLR) aim to prevent catastrophic forgetting.
• Existing SPLR methods rely on point estimates and manual hyperparameter tuning, limiting their efficiency and adaptability.

Purpose of the Study:

• To propose Bayesian Compression for Shared and Private Latent Representations (BCSPLR), a novel continual learning approach.
• To develop a principled framework for learning compact and accurate models in continual learning scenarios.
• To address the limitations of point estimates and hyperparameter tuning in SPLR.

Main Methods:

• Developed Bayesian Compression for SPLR (BCSPLR) using a principled Bayesian framework.
• BCSPLR learns task-specific latent features with significant changes and task-invariant representations with small changes.
• Evaluated BCSPLR on MNIST, CIFAR100, and ImageNet100 datasets.

Main Results:

• BCSPLR successfully learns shared and private compact structures, resulting in fewer parameters.
• The method achieves comparable training times to existing state-of-the-art continual learning algorithms.
• BCSPLR demonstrates excellent model performance, outperforming other continual learning approaches.

Conclusions:

• BCSPLR offers an effective solution for continual learning by enabling compact model structures and preserving accuracy.
• The Bayesian approach overcomes limitations of previous SPLR methods, providing a more robust and efficient continual learning strategy.
• BCSPLR represents a significant advancement in continual learning, achieving strong performance with reduced model complexity.