A continual learning framework with long-term and multiple short-term memory networks

Continual learning aims to sequentially accumulate knowledge while balancing stability and plasticity. Most existing methods focus on mitigating catastrophic forgetting, often at the expense of plasticity. While incorporating an auxiliary memory component to focus on new task data proves effective for enhancing plasticity, using a single network to achieve this may be insufficient to capture the full diversity of knowledge in new tasks. Drawing inspiration from neuroscience research highlighting the value of rich, distributed memory representations, we propose a novel framework that incorporates multiple short-term memory networks to capture diverse task-specific knowledge, alongside a long-term memory network to preserve prior knowledge. To maximize the framework's potential, we identify the limitations of the commonly used Euclidean distance-based regularizers and develop a Gaussian mixture model-based regularizer, encouraging flexible selection and integration of knowledge from the multiple short-term networks. Theoretical analysis and experimental study on various benchmarks demonstrate the efficacy and advantage of our framework in better balancing knowledge retention and new learning. In addition, our framework is compatible with various weight regularization based algorithms and the methods that employ function regularization with replay, providing a versatile approach to enhance continual learning across different paradigms.