Delving Into the Training Dynamics for Image Classification.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks (DNNs) training dynamics (TD) are increasingly explored.
• Current research often uses limited TD quantities, hindering comprehensive understanding and application.
• Effective TD representations are needed to improve DNN training processes.

Purpose of the Study:

• To develop effective TD representations for DNNs.
• To apply these representations to enhance practical learning tasks.
• To identify crucial TD quantities for model training insights.

Main Methods:

• Extracted epoch-wise vectors of 142 TD quantities per sample.
• Designed a self-supervised and supervised learning strategy for deep TD representation learning.
• Developed novel methods for noisy label detection and imbalance learning using deep TD representations.

Main Results:

• Identified neighborhoods and logits as the most important TD quantities, challenging traditional focus on loss and margin.
• Achieved superior performance in noisy label detection and imbalance learning tasks.
• Demonstrated that high-level TD quantities enhance understanding of model training.

Conclusions:

• Deep TD representations offer a more effective approach to understanding and improving DNN training.
• The proposed methods show significant improvements in practical applications like noisy label detection and imbalance learning.
• Neighborhoods and logits are critical TD quantities for effective DNN analysis.