Continual learning with Bayesian compression for shared and private latent representations

This paper proposes a new continual learning method with Bayesian Compression for Shared and Private Latent Representations (BCSPLR), which learns a compact model structure while preserving the accuracy. In Shared and Private Latent Representations (SPLR), task-invariant and task-specific latent representations are efficiently learned to avoid catastrophic forgetting, whereas SPLR produces point estimates of parameters and manually tunes multiple hyperparameters. To overcome these problems, a principle framework is used to develop Bayesian Compression for SPLR, which can learn task-specific latent features with significant changes and task-invariant latent representations with small changes in the continual learning scenarios. To evaluate our algorithm, MNIST, CIFAR100 and ImageNet100 datasets are used to verify our BCSPLR. From these experiments, our model can learn shared and private compact structure which means fewer parameters, and obtain comparable training time, importantly, the model performance is also excellent compared with the state-of-the-art continual learning algorithms.