Self-supervised knowledge distillation for complementary label learning

Area of Science:

• Machine Learning
• Computer Vision

Background:

• Traditional supervised learning requires accurate labels, which are costly and time-consuming to obtain.
• Complementary labels, indicating incorrect classes, offer a more efficient alternative for data labeling.
• Existing methods for complementary label learning often neglect valuable information within the data and models themselves.

Purpose of the Study:

• To propose a novel framework for complementary label learning that leverages self-supervised learning and self-distillation.
• To improve the performance of models trained with complementary labels by utilizing inherent data and model information.
• To address the limitations of current state-of-the-art methods in complementary label learning.

Main Methods:

• Integration of self-supervised learning (rotation and transformation) as an auxiliary task to learn transferable representations.
• Application of entropy regularization to ensure sharper network outputs.
• Utilization of knowledge distillation to transfer "dark knowledge" from a teacher network to a student network.
• Development of a unified framework combining these techniques for complementary learning.

Main Results:

• The proposed method demonstrates significant improvements in accuracy compared to existing state-of-the-art approaches.
• Experiments show the effectiveness of integrating self-supervised learning and self-distillation in complementary label learning.
• The framework successfully extracts and utilizes rich information from data and models for enhanced performance.

Conclusions:

• The novel framework effectively enhances complementary label learning by incorporating self-supervised and self-distillation strategies.
• This approach offers a more efficient and powerful alternative to traditional supervised learning paradigms.
• The findings highlight the potential of leveraging data and model introspection for improved machine learning performance.