Subsampled Hessian Newton Methods for Supervised Learning.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Computational Science

Background:

• Newton methods are effective for supervised learning but computationally expensive for large datasets due to Hessian matrix calculations.
• Subsampled Newton methods approximate the Hessian using data subsets to reduce cost, but can suffer from slower convergence due to less accurate search directions.

Purpose of the Study:

• To enhance the efficiency and effectiveness of subsampled Newton methods for large-scale supervised learning.
• To address the issue of suboptimal search directions in existing subsampled Hessian methods.

Main Methods:

• Proposing novel techniques to improve subsampled Hessian Newton methods.
• Solving a two-dimensional subproblem iteratively to refine search directions.
• Minimizing the second-order approximation of function values more effectively.

Main Results:

• Theoretical convergence of the proposed method is proven.
• Significant reductions in running time observed for the subsampled Hessian Newton method.
• Experimental validation across logistic regression, linear SVM, maximum entropy, and deep networks.

Conclusions:

• The enhanced subsampled Newton method offers a compelling alternative to standard Newton methods for large-scale classification.
• Improved search direction refinement leads to substantial computational speedups.
• The method demonstrates practical applicability and efficiency in various machine learning models.