Continual pre-training mitigates forgetting in language and vision.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Pre-trained models commonly initialize Continual Learning (CL) but are seldom pre-trained *during* CL.
• Investigating Continual Pre-Training (CPT) where models adapt to data streams before downstream task fine-tuning is crucial.

Purpose of the Study:

• To evaluate the Continual Pre-Training scenario and its impact on model forgetting.
• To analyze factors influencing forgetting in CPT, including input modality, architecture, and pre-training protocol.
• To introduce a Sample-Efficient Pre-training (SEP) method to accelerate the pre-training phase.

Main Methods:

• Developed an evaluation protocol for CPT using a Forgetting Control dataset.
• Disentangled the effects of input modality (NLP, Vision), architecture (Transformer, ResNet), and pre-training (supervised, self-supervised) on forgetting.
• Proposed and evaluated the Sample-Efficient Pre-training (SEP) method.

Main Results:

• The pre-training protocol significantly impacts forgetting, more so than other factors.
• Self-supervised CPT in both NLP and Vision effectively mitigates forgetting without specialized CL techniques.
• Factors like model depth, input modality, and architecture type were less critical for reducing forgetting.

Conclusions:

• Self-supervised continual pre-training is a powerful, surprisingly simple strategy to prevent catastrophic forgetting.
• CPT, particularly with self-supervision, offers a robust approach to building adaptable models for non-stationary data streams.