Incremental and Parallel Machine Learning Algorithms With Automated Learning Rate Adjustments.

Area of Science:

• Optimization
• Machine Learning
• Convex Analysis

Background:

• Existing machine learning algorithms for convex optimization exhibit slow convergence due to fixed learning rates.
• Determining optimal learning rates beforehand is a significant challenge, hindering practical applications.

Purpose of the Study:

• To propose two new machine learning algorithms that incorporate a line search method for automatic, run-time learning rate determination.
• To enhance convergence speed and applicability to constrained nonsmooth convex optimization problems.

Main Methods:

• Developed two algorithms: one based on incremental subgradient and another on parallel subgradient, both utilizing a line search method.
• The line search method algorithmically determines appropriate learning rates, satisfying weaker conditions than existing methods.

Main Results:

• The proposed algorithms demonstrate faster convergence and improved performance compared to existing methods in experimental tests.
• These algorithms show comparable or better performance than Pegasos in support vector machine learning tasks regarding accuracy, objective function value, and computation time.
• Successful application in training a multilayer neural network highlights potential in deep learning.

Conclusions:

• The novel algorithms generalize existing subgradient methods for constrained nonsmooth convex optimization.
• The line search approach offers a more robust and efficient way to handle learning rates in machine learning.