Spurious Local Minima Provably Exist for Deep CNNs: Theory and Application

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks, particularly convolutional neural networks (CNNs), often exhibit complex loss landscapes.
• The presence of spurious local minima can hinder the training process and prevent models from reaching optimal performance.

Purpose of the Study:

• To prove the existence of a general family of spurious local minima in CNNs with specific properties.
• To develop a deterministic optimization method to escape these spurious local minima.

Main Methods:

• Construction of spurious local minima by perturbing parameter space and strategically grouping data samples.
• Addressing challenges posed by convolutional layers to ensure targeted perturbation effects.
• Designing a deterministic optimization algorithm based on the spurious local minima construction.

Main Results:

• Demonstrated the general existence of spurious local minima applicable to arbitrary CNN architectures.
• Experimental validation on CIFAR-10, CIFAR-100, and ImageNet-1k datasets confirmed theoretical findings.
• The proposed optimization method consistently outperformed Stochastic Gradient Descent (SGD) and Adam, achieving an average accuracy improvement of 0.27% across architectures.

Conclusions:

• Spurious local minima are a general phenomenon in deep learning models like CNNs.
• The developed optimization technique offers a reliable method to escape these minima and enhance model accuracy.
• The findings have broad applicability to various neural network architectures including CNNs, ResNets, MLPs, and transformers.