Theory of adaptive SVD regularization for deep neural networks

Area of Science:

• Machine Learning
• Deep Learning
• Artificial Intelligence

Background:

• Deep neural networks excel at complex tasks but are prone to overfitting.
• Existing regularization methods lack adaptability to dynamic training conditions.

Purpose of the Study:

• To propose and evaluate an adaptive regularization method based on Singular Value Decomposition (SVD) to mitigate overfitting in deep learning.
• To enhance model performance by dynamically adjusting regularization based on training characteristics.

Main Methods:

• Developed Adaptive SVD Regularization (ASR) by evaluating overfitting using condition numbers of synaptic matrices.
• When overfitting is detected, synaptic matrices are approximated using SVD.
• Incorporated a Tikhonov term to ensure synaptic weights converge to SVD approximations, addressing limitations of pure SVD approximation.

Main Results:

• ASR effectively overcomes overfitting, as demonstrated through visualizations and performance on benchmark datasets (MNIST, F-MNIST, SVHN, CIFAR-10, CIFAR-100).
• Achieved high accuracies: 99.4% (MNIST), 95.7% (F-MNIST), 97.1% (SVHN), 93.2% (CIFAR-10), and 55.6% (CIFAR-100).
• ASR improves overfitting and validation loss with a negligible increase in training time compared to non-regularized learning.

Conclusions:

• Adaptive SVD Regularization (ASR) offers an effective and efficient solution for dynamic overfitting in deep learning models.
• The proposed method demonstrates significant improvements in accuracy and generalization across various datasets and network configurations.
• ASR presents a viable alternative to static regularization techniques, adapting to the evolving nature of the training process.