Learning Student Network Under Universal Label Noise.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Data-free knowledge distillation (DFKD) trains student networks without original data.
• Existing DFKD methods often overlook open-set label noise in web-collected data.
• This limitation hinders student network performance in realistic scenarios.

Purpose of the Study:

• To propose a novel DFKD paradigm addressing universal label noise (closed-set and open-set).
• To enhance student network learning by effectively utilizing web-collected data with mixed label noise.
• To improve the robustness and performance of distilled models.

Main Methods:

• Collected web data with universal label noise.
• Split data into clean, closed-noisy, and open-noisy sets based on prediction uncertainty.
• Refined labels for closed-noisy data using the teacher network.
• Employed noise-robust hybrid contrastive learning on clean and refined closed-noisy sets.
• Utilized self-supervised learning for open-noisy (unlabeled) data.

Main Results:

• The proposed method significantly outperforms existing DFKD approaches on image classification tasks.
• Demonstrated superior performance in handling both closed-set and open-set label noise.
• Achieved effective knowledge transfer from teacher to student networks even with noisy data.

Conclusions:

• The novel DFKD paradigm effectively tackles universal label noise, including the previously ignored open-set noise.
• The hybrid approach of label refinement, contrastive learning, and self-supervised learning enhances distillation.
• This work provides a more realistic and effective solution for data-free knowledge distillation using web data.