Tolerant Self-Distillation for image classification.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Deep neural networks (DNNs) are prone to overfitting, especially with limited training data.
• Overfitting occurs when models learn training data too well, including noise and outliers, leading to poor generalization.
• Existing methods often require large datasets or complex pre-training strategies.

Purpose of the Study:

• To introduce novel metrics from intra-class distribution for analyzing overfitting in DNNs.
• To propose a new knowledge distillation method, Tolerant Self-Distillation (TSD), to alleviate overfitting.
• To enable effective knowledge distillation without the need for a pre-trained teacher model.

Main Methods:

• Developed two metrics based on the intra-class distribution of correctly and incorrectly predicted samples.
• Proposed Tolerant Self-Distillation (TSD), an online knowledge distillation approach.
• TSD utilizes an online updating memory to store past predictions, distilling knowledge across iterations by using incorrect predictions to supervise correct ones and vice versa.

Main Results:

• The proposed TSD method effectively mitigates the overfitting issue in deep neural networks.
• TSD alleviates premature convergence caused by over-confident predictions, leading to better local optima.
• Experiments on various image classification benchmarks (small-scale, large-scale, fine-grained) demonstrate TSD's superiority.

Conclusions:

• The novel intra-class distribution metrics offer a new perspective on understanding overfitting.
• Tolerant Self-Distillation (TSD) provides an effective and efficient approach to combat overfitting in DNNs.
• TSD shows significant improvements in model generalization across diverse datasets.