Delving Into the Training Dynamics for Image Classification

In recent years, there has been an increase in exploring and applying the training dynamics (TD) of deep neural networks (DNNs). Current studies typically rely on quite limited TD quantities and apply their sequences to understand or aid training. This study investigates how to create more effective TD representations, and then apply them to improve the training process of real learning tasks. Specifically, first, an epoch-wise vector comprising 142-dimensional TD quantities, such as loss, is extracted for each sample. Second, a new learning strategy with both self-supervised and supervised learning is designed to learn the deep TD representation of each sample on 200 typical image classification tasks. Third, two novel methods for noisy label detection and imbalance learning, respectively, are presented based on deep TD representations. Our study reveals that neighborhoods and logits are the most important TD quantities, unlike the traditional research that focuses on loss and margin. Moreover, our method based on deep TD representations achieves better performance and demonstrates that high-level TD quantities can facilitate understanding model training, leading to improvements in practical learning tasks, such as noisy label detection and imbalance learning. All the codes are available at https://github.com/limengyang1992/TD_Exploring.