Taming Overconfident Prediction on Unlabeled Data From Hindsight.

Area of Science:

• Machine Learning
• Artificial Intelligence
• Computer Vision

Background:

• Semi-supervised learning (SSL) relies on minimizing prediction uncertainty on unlabeled data for optimal performance.
• Current methods for distilling low-entropy predictions in SSL are often heuristic and lack informativeness.
• Prediction uncertainty is commonly quantified using entropy derived from transformed output probabilities.

Purpose of the Study:

• To propose a novel dual mechanism, adaptive sharpening (ADS), for more effective distillation in SSL.
• To address the limitations of existing heuristic distillation strategies in semi-supervised learning.
• To enhance the performance of state-of-the-art SSL methods through improved distillation.

Main Methods:

• Developed adaptive sharpening (ADS), a dual mechanism involving soft-thresholding and sharpening of predictions.
• Adaptively masks out determinate and negligible predictions while sharpening informed ones.
• Theoretically analyzed ADS and compared its traits against various existing distillation strategies.

Main Results:

• ADS significantly improves the performance of state-of-the-art semi-supervised learning methods.
• Experimental results validate the effectiveness of ADS as a plug-in module.
• The proposed method demonstrates superior performance compared to traditional distillation strategies.

Conclusions:

• Adaptive sharpening (ADS) offers a more informative and effective approach to distillation in SSL.
• ADS provides a robust and adaptable solution for enhancing SSL model training.
• This work lays a foundation for future research in distillation-based semi-supervised learning.