Impact of Noisy Supervision in Foundation Model Learning

Area of Science:

• Deep Learning
• Machine Learning
• Artificial Intelligence

Background:

• Foundation models are typically pre-trained on large datasets, then fine-tuned for specific tasks.
• Pre-training datasets can contain label noise, potentially compromising model generalization and introducing risks.
• The impact of pre-training noise on downstream task performance remains inadequately understood.

Purpose of the Study:

• To comprehensively analyze the nature of noise in pre-training datasets.
• To investigate the effects of pre-training noise on both in-domain and out-of-domain generalization.
• To propose and validate a method for mitigating noise impacts in foundation models.

Main Methods:

• Extensive experiments using synthetic noisy datasets (ImageNet-1K, YFCC15M, CC12M) for supervised and contrastive pre-training.
• Analysis of noise impact across various dataset scales, noise types, model architectures, and pre-training objectives.
• Development and evaluation of NMTune, a novel tuning method for feature space adaptation.

Main Results:

• Slight pre-training noise can benefit in-domain performance but consistently degrades out-of-domain performance.
• These findings are robust across different dataset scales, noise types, model architectures, and pre-training objectives.
• NMTune effectively mitigates noise effects and enhances generalization in both parameter-efficient and black-box tuning scenarios.

Conclusions:

• Pre-training noise fundamentally alters feature space representations, impacting model generalization.
• The proposed NMTune method offers a practical solution for improving the robustness of foundation models trained on noisy data.
• This research establishes 'Noisy Model Transfer Learning' as a critical area for future investigation.