Low Dimensional Trajectory Hypothesis is True: DNNs Can Be Trained in Tiny Subspaces.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Deep Learning

Background:

• Deep neural networks (DNNs) possess massive parameters, often exhibiting redundancy.
• The hypothesis suggests DNNs can be effectively trained within low-dimensional subspaces.

Purpose of the Study:

• To introduce Dynamic Linear Dimensionality Reduction (DLDR) for optimizing DNNs.
• To leverage the low-dimensional properties of DNN training trajectories.

Main Methods:

• Dynamic Linear Dimensionality Reduction (DLDR) technique.
• Development of an efficient quasi-Newton-based optimization algorithm.
• Experimental validation of DNN training in reduced dimensional spaces.

Main Results:

• DNNs optimized in 40-dimensional spaces achieve performance comparable to standard training.
• The DLDR method demonstrates efficiency and robustness to label noise.
• Improved performance observed in well-trained models using the proposed subspace optimization.

Conclusions:

• DLDR offers an efficient approach to train deep neural networks in low-dimensional subspaces.
• The method significantly reduces computational complexity while maintaining high performance.
• DLDR provides a promising direction for more efficient deep learning model training.