Neural Network Training With Levenberg-Marquardt and Adaptable Weight Compression

Area of Science:

• Artificial Intelligence
• Machine Learning
• Neural Networks

Background:

• Neural network training frequently encounters the flat-spot problem, characterized by diminishing gradients that impede effective weight updates.
• First-order algorithms can mitigate this by normalizing activations, but second-order algorithms face computational challenges with additional parameters.

Purpose of the Study:

• To introduce a novel algorithm, Levenberg-Marquardt with weight compression (LM-WC), to address the flat-spot problem in neural network training.
• To enhance neural network convergence without requiring additional learned parameters.

Main Methods:

• The proposed LM-WC algorithm compresses neuron weights to shift activations from saturated to linear regions.
• It incorporates an adaptable compression parameter to prevent training failures and boost convergence probability.
• The method avoids the computational overhead of additional learned parameters inherent in some other approaches.

Main Results:

• Experiments demonstrated LM-WC's superior performance against standard Levenberg-Marquardt (LM) and LM with random restarts.
• The LM-WC algorithm showed a significant improvement in training success rates, exceeding tenfold in some cases.
• Performance was validated across various network architectures and benchmark datasets.

Conclusions:

• LM-WC effectively resolves the flat-spot problem in neural network training.
• The algorithm offers a computationally efficient solution for improving convergence and training success.
• LM-WC presents a promising advancement for training complex neural network models.