HiDe-PET: Continual Learning via Hierarchical Decomposition of Parameter-Efficient Tuning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Pre-trained models (PTMs) are crucial for continual learning (CL), offering knowledge transfer and preventing catastrophic forgetting.
• Parameter-efficient tuning (PET) methods, particularly prompt-based approaches, are used with frozen PTM backbones for CL, but often yield suboptimal results.

Purpose of the Study:

• To propose a unified framework for CL with PTMs and PET that offers theoretical and empirical advancements.
• To address the sub-optimal performance of current prompt-based PET techniques in CL settings.

Main Methods:

• Performed theoretical analysis of the CL objective in a pre-training context, decomposing it into within-task prediction, task-identity inference, and task-adaptive prediction.
• Introduced Hierarchical Decomposition PET (HiDe-PET), optimizing the decomposed objective by incorporating task-specific and shared knowledge via PET techniques and recovering pre-trained representations.
• Investigated the impact of implementation strategy, PET technique, PET architecture, and adaptive knowledge accumulation under distribution shifts.

Main Results:

• HiDe-PET demonstrates superior performance across various CL scenarios compared to strong baselines.
• The framework provides insights into the distinct impacts of implementation strategies, PET techniques, and architectures in CL.
• Adaptive knowledge accumulation is shown to be effective amidst significant distribution changes.

Conclusions:

• The proposed HiDe-PET framework offers a theoretically grounded and empirically effective approach for continual learning with pre-trained models and parameter-efficient tuning.
• HiDe-PET advances the field by optimizing decomposed objectives and enhancing knowledge transfer and resilience in sequential learning tasks.