Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) are powerful but struggle with noisy or unlabeled data.
• Current training methods can be computationally expensive and may not generalize well.
• Understanding deep learning dynamics, like double-descent and self-supervised learning collapse, remains a challenge.

Purpose of the Study:

• To propose a unified, computationally efficient training algorithm for DNNs.
• To enhance both supervised and self-supervised learning by leveraging model predictions.
• To analyze and explain phenomena in deep learning training dynamics.

Main Methods:

• Developed self-adaptive training, a unified algorithm that dynamically calibrates training using model predictions.
• Analyzed training dynamics of DNNs on corrupted datasets (e.g., noise, adversarial examples).
• Validated the approach on CIFAR, STL, and ImageNet datasets across classification and representation learning tasks.

Main Results:

• Self-adaptive training improves DNN generalization under label noise and enhances self-supervised representation learning.
• Model predictions were shown to magnify useful data information, even without labels.
• The method offers insights into the double-descent phenomenon and self-supervised learning collapse.

Conclusions:

• Self-adaptive training is an effective and computationally efficient method for improving DNNs.
• Model predictions offer a valuable, low-cost mechanism for enhancing deep learning.
• The approach advances understanding of deep learning training dynamics and offers practical benefits.