Non-differentiable saddle points and sub-optimal local minima exist for deep ReLU networks

Area of Science:

• Deep Learning
• Optimization Theory
• Computational Mathematics

Background:

• The performance of deep neural networks is significantly influenced by the characteristics of their loss landscape, particularly the presence of local minima and saddle points.
• Understanding these landscape features is crucial for developing effective optimization algorithms in deep learning.

Purpose of the Study:

• To theoretically investigate the existence of non-differentiable sub-optimal local minima and saddle points in deep Rectified Linear Unit (ReLU) networks of arbitrary depth.
• To analyze the implications of these landscape features on optimization algorithms used for training deep learning models.

Main Methods:

• Theoretical analysis of the loss surface for deep ReLU networks.
• Mathematical proofs for the existence of non-differentiable saddle points and sub-optimal local minima under specific loss functions (squared loss, cross-entropy loss).
• Empirical validation using experiments on both real-world and synthetic datasets.

Main Results:

• Existence of non-differentiable saddle points is proven for deep ReLU networks with squared loss or cross-entropy loss under standard assumptions.
• Non-differentiable sub-optimal local minima are proven to exist in deep ReLU networks with cross-entropy loss when certain data distribution conditions are met.
• Experimental results align with theoretical predictions, confirming the presence of these critical points in the loss landscape.

Conclusions:

• The loss landscape of deep ReLU networks is characterized by the guaranteed existence of non-differentiable saddle points and, under certain conditions, sub-optimal local minima.
• These findings provide theoretical insights into the challenges faced by optimization algorithms in deep learning and suggest avenues for algorithm improvement.
• The study underscores the importance of considering the complex geometry of the loss landscape for effective deep neural network training.