Training a Two-Layer ReLU Network Analytically.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Neural Network Optimization

Background:

• Neural networks commonly use gradient descent variants like stochastic gradient descent and Adam.
• Theoretical work suggests not all critical points in two-layer ReLU networks are local minima.
• Existing methods may struggle to find optimal solutions efficiently.

Purpose of the Study:

• To explore an alternative algorithm for training two-layer neural networks with ReLU-like activation and square loss.
• To investigate if analytical layer-wise optimization can outperform gradient descent methods.
• To assess the speed and parameter tuning requirements of the proposed algorithm.

Main Methods:

• Developed an algorithm that analytically optimizes one layer of a two-layer neural network while keeping the other fixed.
• Maintained the neuron activation pattern during the analytical optimization process.
• Utilized square loss for training the neural networks.

Main Results:

• The proposed algorithm found deeper optima compared to stochastic gradient descent and Adam.
• Significantly smaller training loss values were achieved on four out of five real datasets.
• The method demonstrated faster training times than gradient descent-based approaches.
• The algorithm requires virtually no tuning parameters.

Conclusions:

• The analytical layer-wise optimization approach offers a promising alternative to standard gradient descent for training neural networks.
• This method achieves superior performance in terms of solution quality and training speed.
• Its simplicity and minimal parameter tuning make it a practical choice for certain neural network training tasks.