Continuous Review and Timely Correction: Enhancing the Resistance to Noisy Labels via Self-Not-True and Class-Wise...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Deep Learning

Background:

• Deep neural networks are powerful but prone to overfitting, especially with mislabeled data.
• The memorization effect causes networks to learn correct labels first, then incorrect ones.
• Early stopping mitigates overfitting but doesn't fully prevent adaptation to incorrect labels or correct errors.

Purpose of the Study:

• To develop a mechanism for continuous review and correction of learned knowledge in deep neural networks.
• To address the challenge of overfitting and memorization caused by mislabeled data.
• To improve the robustness of deep learning models in the presence of label noise.

Main Methods:

• Introduced self-not-true-distillation (SNTD), a novel method using self-distillation to review and reinforce correct labels.
• SNTD masks true labels during distillation, focusing on correcting erroneous knowledge from non-true classes.
• Proposed SNTD+, incorporating class-wise distillation and dynamic weight adjustment for improved performance.

Main Results:

• SNTD enables networks to repeatedly revisit and reinforce accurate information while correcting inaccuracies from mislabeled data.
• SNTD+ enhances robustness by selecting class-specific teacher networks and dynamically adjusting guidance weights.
• The proposed methods show significant improvements in tackling complex scenarios with label noise.

Conclusions:

• Continuous review and correction mechanisms are essential for deep neural networks facing label noise.
• SNTD and SNTD+ offer effective strategies to mitigate the memorization effect and improve model accuracy.
• The class-wise and dynamic adjustments in SNTD+ are crucial for handling diverse learning trajectories and varying teacher influence.