A topological description of loss surfaces based on Betti Numbers

Area of Science:

• Deep Learning
• Computational Topology
• Machine Learning

Background:

• Understanding the loss function surface is crucial for optimizing deep learning models trained with gradient descent.
• Identifying spurious minima and characterizing gradient dynamics are key challenges in deep learning research.

Purpose of the Study:

• To introduce a novel topological measure for quantifying the complexity of loss functions in multilayer neural networks.
• To analyze how architectural choices and training parameters influence loss topology.

Main Methods:

• Derivation of upper and lower bounds for loss function complexity in deep and shallow neural networks.
• Comparison of architectures with sigmoidal activation functions.
• Analysis of the impact of hidden unit count, training models, and activation functions.

Main Results:

• Established how loss complexity is influenced by the number of hidden units, training models, and activation functions.
• Demonstrated that certain architectural changes, like L2 regularization or skip connections, may not alter loss topology in specific scenarios.
• Provided a comparative analysis of deep versus shallow architectures.

Conclusions:

• The proposed topological measure offers a new perspective on understanding loss function landscapes in deep learning.
• Loss function complexity is demonstrably affected by network architecture and training parameters.
• Specific modifications to model architecture do not always impact the fundamental topology of the loss surface.