Enhancing progressive ensemble learning via normalized extra-Gradient initialization

Deep learning models continue to demand significant computational resources as their scale grows. Progressive training, which incrementally enlarges models during training, offers an effective remedy. Among its most popular applications is progressive ensemble learning, a method trains ensembles of models in a stage-wise manner. To make progressive ensemble learning more efficient, a copy initialization method has been proposed to initialize newly added modules by duplicating parameters from earlier models. Nevertheless, copy initialization compromises training stability, and few works focus on designing novel initialization methods based on learning theory. To bridge this gap, we formulate progressive ensemble learning as a functional optimization problem. Building upon this theoretical framework, we introduce Normalized Extra-Gradient (NEG) Initialization, a novel method designed to enhance the performance of progressive ensemble learning. We deliver rigorous theoretical insights, including convergence guarantees and an Edge-of-Stability analysis, which explain how NEG Initialization benefits both enhancing training efficiency and stability. To validate the superiority of our method, we conduct proof-of-concept experiments on synthetic data, and apply progressive ensemble learning to Vision Transformer (ViT) models on ImageNet-200 and ImageNet-1K. Extensive empirical results highlight the advantages of our approach over baselines and underscore its broad applicability.