Semi-supervised Learning with Online Knowledge Distillation for Skin Lesion Classification

Deep learning has emerged as a promising approach for skin lesion analysis. However, existing methods mostly rely on fully supervised learning, requiring extensive labeled data, which is challenging and costly to obtain. To alleviate this annotation burden, this study introduces a novel semi-supervised deep learning approach that integrates ensemble learning with online knowledge distillation for enhanced skin lesion classification. Our methodology involves training an ensemble of convolutional neural network models, using online knowledge distillation to transfer insights from the ensemble to its members. This process aims to enhance the performance of each model within the ensemble, thereby elevating the overall performance of the ensemble itself. Post-training, any individual model within the ensemble can be deployed at test time, as each member is trained to deliver comparable performance to the ensemble. This is particularly beneficial in resource-constrained environments. Experimental results demonstrate that the knowledge-distilled individual model performs better than independently trained models. Our approach outperforms current state-of-the-art on ISIC (International Skin Imaging Collaboration) 2018, ISIC 2019, and ISIC 2020 benchmark datasets. For example, with only 10% labeled data on ISIC 2018, we observe around $3\%$ gain in macro F1 versus ReFixMatch-LS, a recently proposed state-of-the-art for skin lesion classification framework. Furthermore, the proposed method significantly reduces label requirements: while a fully supervised baseline reaches an F1 of $62.40\pm 0.71$ with 20% labeled data, our approach attains comparable performance using just 10%. These results highlight its superior label efficiency and practical relevance for real-world skin lesion classification. The code is available online.