Complementary label learning based on knowledge distillation.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Science

Background:

• Complementary Label Learning (CLL) is a weakly supervised method using non-target class information.
• Existing CLL methods often underutilize supervisory signals within complementary labels.
• Enhancing these signals is crucial for improving CLL performance.

Purpose of the Study:

• To propose a novel framework, Complementary Label Enhancement based on Knowledge Distillation (KDCL), to enrich supervision in CLL.
• To address the limitations of current CLL approaches in leveraging complementary label information.
• To improve the accuracy of models trained with complementary labels.

Main Methods:

• Introduced KDCL, a framework employing a teacher-student deep neural network architecture.
• The teacher model softens complementary labels to enhance supervision.
• The student model learns from these softened labels, with both trained on complementary-labeled data.

Main Results:

• KDCL-optimized models demonstrated superior accuracy compared to baseline CLL methods across four datasets (MNIST, F-MNIST, K-MNIST, CIFAR-10).
• Experiments utilized diverse datasets and teacher-student model pairs (Lenet-5+MLP, DenseNet-121+ResNet-18).
• The proposed method effectively improved classification performance.

Conclusions:

• KDCL offers a significant advancement in Complementary Label Learning by effectively enhancing supervision.
• The knowledge distillation approach within KDCL successfully enriches complementary label information.
• This framework presents a promising direction for improving weakly supervised learning tasks.