Recursion Newton-Like Algorithm for l_2,0-ReLU Deep Neural Networks.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Rectified Linear Unit deep neural networks (ReLU-DNNs) are powerful but suffer from high memory and computational demands due to numerous parameters.
• l2,0 regularization is a technique to address the parameter overload in deep learning models.

Purpose of the Study:

• To develop a novel algorithm for simultaneous training and compression of ReLU-DNNs.
• To address the computational burden and memory requirements of large ReLU-DNN models.

Main Methods:

• A recursion Newton-like algorithm (RNLA) was designed to train and compress ReLU-DNNs using l2,0 regularization.
• The training model was reformulated into a constrained optimization problem with network nodes as variables.
• Minimization subproblems were derived, and their first-order optimality conditions were used to obtain piecewise linear matrix equations.
• A column Newton-like method was employed to solve these equations efficiently.

Main Results:

• The proposed RNLA effectively trains and compresses ReLU-DNNs.
• Numerical experiments on real datasets confirmed the method's effectiveness and applicability.
• The algorithm demonstrates a lower computational scale and cost compared to existing methods.

Conclusions:

• The developed RNLA provides an effective solution for training and compressing ReLU-DNNs.
• The method successfully mitigates the memory and computation issues associated with large deep neural networks.
• RNLA is a practical and applicable technique for optimizing deep learning models.