An Efficient Fisher Matrix Approximation Method for Large-Scale Neural Network Optimization.

Area of Science:

• Machine Learning
• Optimization Algorithms
• Numerical Analysis

Background:

• First-order optimization methods are widely used for large-scale empirical risk minimization.
• Second-order methods offer faster convergence but often face computational challenges due to large matrix inversions.

Purpose of the Study:

• To develop an efficient second-order optimization method for large-scale problems.
• To reduce the computational burden associated with second-order methods by proposing a novel matrix approximation.

Main Methods:

• Proposed a second-order method utilizing parameters in the real matrix space and a matrix-product approximate Fisher matrix (MatFisher).
• Employed matrix delayed update and block diagonal approximation techniques to control computational cost.
• Analyzed global and superlinear local convergence properties.

Main Results:

• The proposed MatFisher significantly reduces the size of the matrix to be inverted compared to traditional Fisher information matrices.
• Computational cost is comparable to first-order methods.
• Demonstrated competitive performance on image classification (ResNet50), quantum chemistry (SchNet), and data-driven PDEs (PINN).

Conclusions:

• The novel second-order method offers an efficient and computationally feasible alternative for large-scale optimization.
• The approach achieves competitive results across diverse scientific and engineering applications.