Enhancing progressive ensemble learning via normalized extra-Gradient initialization.

Area of Science:

• Deep Learning
• Machine Learning
• Computer Vision

Background:

• Deep learning models require substantial computational resources due to increasing scale.
• Progressive training incrementally enlarges models, offering a solution to resource demands.
• Progressive ensemble learning trains model ensembles stage-wise, but copy initialization methods compromise stability.

Purpose of the Study:

• To address the training stability issues in progressive ensemble learning.
• To develop a novel initialization method grounded in learning theory.
• To improve the efficiency and stability of progressive ensemble learning.

Main Methods:

• Formulated progressive ensemble learning as a functional optimization problem.
• Introduced Normalized Extra-Gradient (NEG) Initialization.
• Provided theoretical insights, including convergence guarantees and Edge-of-Stability analysis.

Main Results:

• NEG Initialization enhances both training efficiency and stability.
• Proof-of-concept experiments on synthetic data validated the method.
• Application to Vision Transformer (ViT) models on ImageNet-200 and ImageNet-1K demonstrated superiority over baselines.

Conclusions:

• NEG Initialization offers a theoretically grounded and empirically validated approach to improve progressive ensemble learning.
• The method enhances computational efficiency and training stability.
• Demonstrated broad applicability, particularly for Vision Transformer models in computer vision tasks.