Continual Unsupervised Generative Modeling

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Variational Autoencoders (VAEs) excel at single-task learning but struggle with continuous learning across domains.
• Catastrophic forgetting, a common issue in machine learning, leads to information loss when VAEs learn new tasks sequentially.

Purpose of the Study:

• To address catastrophic forgetting in VAEs during continual learning.
• To develop a theoretical framework and practical methods for preserving knowledge across sequential tasks.

Main Methods:

• Derived a theoretical upper bound for negative sample log-likelihood in continual learning.
• Introduced the Dynamic Expansion Graph Model (DEGM) to optimize model size and promote positive knowledge transfer.
• Proposed the Dynamic Expansion Graph Adaptive Mechanism (DEGAM) to regulate graph structure and enhance knowledge transfer.

Main Results:

• The theoretical framework provides insights into network forgetting behavior.
• DEGM and DEGAM dynamically build and adapt graph structures for improved learning.
• Experimental results demonstrate superior performance compared to existing continual learning baselines.

Conclusions:

• The proposed methodology effectively mitigates catastrophic forgetting in VAEs.
• Dynamic graph structures and adaptive mechanisms enhance positive knowledge transfer in continual learning settings.
• This approach offers a promising solution for VAEs in sequential task learning scenarios.