Block-cyclic stochastic coordinate descent for deep neural networks.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Computer Vision

Background:

• Stochastic coordinate descent methods are widely used for large-scale optimization.
• Existing methods can be sensitive to outliers in training data, affecting generalization.
• Improving the efficiency and robustness of optimization algorithms is crucial for deep learning.

Purpose of the Study:

• To introduce a novel optimization algorithm, block-cyclic stochastic coordinate descent (BCSC).
• To enhance the generalization capabilities and accuracy of machine learning models, particularly in image classification.
• To mitigate the impact of outliers during the training process.

Main Methods:

• Developed BCSC, a stochastic first-order optimization algorithm.
• Incorporated a cyclic constraint for data and parameter selection in stochastic block-coordinate descent.
• Utilized distinct data subsets for updating distinct parameter subsets to reduce outlier effects.

Main Results:

• BCSC demonstrated superior generalization performance compared to state-of-the-art optimization methods.
• Achieved higher accuracy in image classification benchmarks within the same number of update iterations.
• Improvements were consistent across various architectures and datasets.

Conclusions:

• BCSC offers a robust and efficient optimization strategy for machine learning.
• The algorithm effectively limits the detrimental effect of outliers, leading to better model generalization.
• BCSC's compatibility with other training techniques allows for flexible integration into existing workflows.