The training process of many deep networks explores the same low-dimensional manifold

Area of Science:

• Machine Learning
• Deep Learning
• Information Geometry

Background:

• Deep neural networks (DNNs) are powerful tools in artificial intelligence.
• Understanding the internal dynamics of DNN training is crucial for improving model performance and reliability.
• Existing methods often struggle to capture the complex, high-dimensional nature of the training process.

Purpose of the Study:

• To apply information-geometric techniques to analyze deep network prediction trajectories during training.
• To investigate the intrinsic dimensionality of the space explored by DNN predictions.
• To determine the influence of various training factors on the exploration of this space.

Main Methods:

• Development of information-geometric methods for analyzing high-dimensional probabilistic models.
• Examination of prediction trajectories of deep networks across diverse architectures and training configurations.
• Comparative analysis of manifold exploration under varied optimization, regularization, data augmentation, and weight initialization strategies.

Main Results:

• Deep network training consistently explores an effectively low-dimensional manifold in the prediction space.
• This shared manifold is observed across networks of varying architectures, sizes, and training methodologies.
• While architectures exhibit distinguishable trajectories, other factors like size and initialization have minimal impact on manifold convergence.

Conclusions:

• The training of deep networks, irrespective of specific configurations, converges to a common low-dimensional manifold.
• Network architecture influences the trajectory but not the fundamental manifold of exploration.
• This finding offers a unified perspective on deep learning training dynamics, simplifying our understanding of complex models.