VPT-NSP²++: Importance-Aware Visual Prompt Tuning in Null Space for Continual Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Continual learning (CL) is crucial for AI models to adapt to dynamic environments and prevent catastrophic forgetting.
• Vision Transformer (ViT) models and visual prompt tuning (VPT) are increasingly popular in CL.
• Existing orthogonal projection methods face challenges when applied to ViTs due to non-linearities.

Purpose of the Study:

• To develop a theoretically guaranteed CL method for ViT models using VPT.
• To adapt orthogonal projection techniques for ViTs, addressing self-attention and LayerNorm complexities.
• To enhance long-term CL performance and improve the stability-plasticity trade-off.

Main Methods:

• Proposed two orthogonality conditions for prompt gradient orthogonal projection in ViTs.
• Introduced an importance-aware orthogonal regularization framework to balance model capacity and plasticity.
• Employed a null-space-based approximation for efficient orthogonal projection implementation.

Main Results:

• The proposed method achieves state-of-the-art performance on class-incremental learning benchmarks.
• Demonstrated enhanced stability and plasticity in long-sequence CL scenarios.
• Effectively addressed challenges of applying orthogonal projection to ViTs.

Conclusions:

• The importance-aware orthogonal regularization framework offers a robust solution for ViT-based CL.
• The method provides theoretical guarantees for stability while improving adaptability.
• This work advances the field of continual learning for complex deep learning models.