Improved Linear Convergence of Training CNNs With Generalizability Guarantees: A One-Hidden-Layer Case

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Convolutional Neural Networks (CNNs) are crucial for image processing.
• Training CNNs involves minimizing complex, often non-convex, loss functions.
• Rectified Linear Unit (ReLU) activation introduces non-smoothness, complicating optimization.

Purpose of the Study:

• To analyze the parameter estimation problem for one-hidden-layer CNNs with ReLU activation.
• To investigate the convergence properties of accelerated gradient descent (GD) for this non-convex problem.
• To establish the theoretical sample complexity for training these networks.

Main Methods:

• Utilized a model estimation perspective, assuming training data is generated by a noisy neural network.
• Applied accelerated gradient descent (GD) with tensor initialization.
• Analyzed convergence rates and established sample complexity bounds for finite Gaussian-distributed datasets.

Main Results:

• Proved that accelerated GD converges to ground-truth parameters (within noise level) at a linear rate.
• Demonstrated that accelerated GD converges faster than vanilla GD.
• Established the first theoretical sample complexity for gradient-based CNN training with ReLU activation.

Conclusions:

• Accelerated GD is effective for optimizing non-convex CNN learning problems, achieving global optima.
• This study provides the tightest bounds on estimation error concerning output noise for such networks.
• The findings offer significant theoretical insights into training non-smooth, non-convex neural networks.