Dynamics in Deep Classifiers Trained with the Square Loss: Normalization, Low Rank, Neural Collapse, and...

Area of Science:

• Machine Learning
• Deep Learning Theory
• Optimization

Background:

• Overparameterized deep networks trained with square loss exhibit complex dynamics.
• Understanding convergence properties and error bounds is crucial for network performance.
• Existing bounds for dense networks do not fully capture the benefits of specific architectures like CNNs.

Purpose of the Study:

• To analyze the properties of training deep homogeneous rectified linear unit networks under square loss.
• To derive novel norm-based error bounds for convolutional layers.
• To investigate the generalization properties of solutions obtained via stochastic gradient descent with weight decay.

Main Methods:

• Modeling gradient flow dynamics under square loss.
• Utilizing Lagrange multipliers and weight decay with gradient descent variants.
• Deriving norm-based bounds for convolutional and dense layers.
• Analyzing quasi-interpolating solutions from stochastic gradient descent.

Main Results:

• Achieved convergence to a minimum Frobenius norm product (ρ) with normalized weight decay.
• Derived significantly improved norm-based bounds for convolutional layers compared to dense networks.
• Demonstrated that stochastic gradient descent with weight decay introduces a bias towards low-rank weight matrices, enhancing generalization.
• Predicted and experimentally verified inherent stochastic gradient descent noise and neural collapse phenomena.
• Showcased the advantage of deep networks for problems suited to sparse architectures.

Conclusions:

• Deep networks, particularly sparse architectures like CNNs, offer advantages for specific problems by avoiding the curse of dimensionality.
• The derived bounds and analysis provide theoretical insights into the generalization capabilities of deep learning models.
• The study confirms the effectiveness of weight decay and gradient descent in achieving desirable network properties.