Stagewise Training With Exponentially Growing Training Sets.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Big Data Analytics

Background:

• Large-scale machine learning training is computationally intensive.
• Existing optimization strategies offer acceleration but further improvements are sought.
• The impact of training data size on optimization efficiency is a key research area.

Purpose of the Study:

• To develop a novel technique for accelerating large-scale machine learning training.
• To investigate the effectiveness of a stagewise training approach with exponentially growing datasets.
• To analyze the compatibility and performance of the proposed method with existing optimization algorithms.

Main Methods:

• Proposed a stagewise training technique (STEGS) that exponentially increases training set size.
• Demonstrated compatibility with proximal gradient descent and gradient hard thresholding (GHT) methods.
• Analyzed the influence of the data growth rate on computational complexity.

Main Results:

• STEGS significantly reduces overall computational complexity.
• The method maintains or improves statistical accuracy compared to standard GHT.
• Applied to large-scale real-world datasets using $l_{2,1}$- and $l_{0}$-norms, STEGS demonstrated practical benefits.
• The data growth rate impacts overall complexity, as analyzed.

Conclusions:

• The stagewise training via exponentially growing the size of the training sets (STEGS) framework offers a promising approach for accelerating large-scale machine learning.
• STEGS provides a significant reduction in computational complexity while preserving or enhancing model accuracy.
• The framework is versatile and effective across various optimization techniques and real-world datasets.