Approximate Fisher Information Matrix to Characterize the Training of Deep Neural Networks

Area of Science:

• Deep Learning
• Computer Vision
• Machine Learning

Background:

• Deep learning models like ResNets and DenseNet are crucial for image classification.
• Understanding the impact of mini-batch size and learning rate on convergence and generalization is vital.
• Current methods for characterizing deep learning performance can be computationally intensive.

Purpose of the Study:

• To introduce a novel methodology for characterizing deep learning network performance.
• To enable practitioners to monitor and control training convergence and generalization.
• To develop an optimized dynamic sampling training approach.

Main Methods:

• Utilizing novel measurements derived from eigenvalues of the approximate Fisher information matrix.
• Efficient computation of these measurements for high-capacity deep models.
• Implementing a dynamic sampling training approach that adjusts mini-batch size and learning rate automatically.

Main Results:

• The proposed measurements effectively characterize deep learning performance.
• The dynamic sampling approach optimizes the training process.
• The dynamic sampling method achieves faster training times and competitive accuracy.
• Demonstrated ability to actively tune mini-batch size and learning rate for improved outcomes.

Conclusions:

• The novel methodology provides insights into deep learning training dynamics.
• Dynamic sampling training offers an efficient and effective approach for image classification.
• This work facilitates better control over deep learning model training and performance.