EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Modern computer vision backbones achieve high performance but require costly training.
• Curriculum learning typically involves training with easier-to-harder data samples.

Purpose of the Study:

• To generalize curriculum learning by reformulating it as a soft-selection function that uncovers patterns within examples.
• To reduce the computationally expensive training time for computer vision models.

Main Methods:

• Proposed a curriculum learning approach that uses all training data at each stage, initially focusing on easier patterns.
• Implemented this by cropping Fourier spectrum inputs for lower-frequency components and modulating data augmentation intensity.
• Developed tailored searching algorithms for curriculum learning schedules and efficient deployment techniques.

Main Results:

• EfficientTrain++ reduces training time for various models (ResNet, ConvNeXt, DeiT, PVT, Swin, CSWin, CAFormer) by [Formula: see text] on ImageNet-1 K/22 K.
• The method achieves this reduction without sacrificing model accuracy.
• Demonstrated efficacy in self-supervised learning tasks, such as Masked Autoencoders (MAE).

Conclusions:

• EfficientTrain++ offers a simple, general, and effective solution for accelerating computer vision model training.
• The approach successfully generalizes curriculum learning by focusing on pattern difficulty within data.
• This method presents a practical and efficient way to optimize training for various deep learning models.