Knowledge Distillation Meets Label Noise Learning: Ambiguity-Guided Mutual Label Refinery

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge distillation (KD) transfers knowledge from large teacher models to smaller student models.
• Existing KD methods often struggle with real-world datasets containing noisy labels, leading to performance degradation.
• Handling noisy labels is crucial for robust and practical deep learning applications.

Purpose of the Study:

• To address the challenge of knowledge distillation with noisy labels.
• To propose a novel method, ambiguity-guided mutual label refinery knowledge distillation (AML-KD), for training student models under label noise.
• To develop a robust framework that improves student model accuracy despite label imperfections.

Main Methods:

• A two-stage label refinery framework is proposed, leveraging a pretrained teacher model.
• Stage one involves label propagation (LP) with small-loss selection guided by the teacher model.
• Stage two employs mutual LP between teacher and student models, incorporating an ambiguity-aware weight estimation (AWE) module to handle ambiguous samples.

Main Results:

• AML-KD demonstrates effectiveness in training accurate and efficient student models even with significant label noise.
• Experimental results on synthetic and real-world noisy datasets show superior performance compared to state-of-the-art KD and label noise learning (LNL) methods.
• The AWE module successfully mitigates overfitting issues caused by ambiguous samples during label refinery.

Conclusions:

• AML-KD offers a robust solution for knowledge distillation in the presence of noisy labels.
• The proposed method enables the creation of high-accuracy, low-cost student models from imperfect data.
• AML-KD advances the field of knowledge distillation by providing a practical approach for real-world noisy datasets.