Data-Efficient Sensor Upgrade Path Using Knowledge Distillation.

Area of Science:

• Computer Vision
• Machine Learning
• Sensor Technology

Background:

• Deep neural networks excel in image classification but require large datasets.
• Transitioning to new sensor modalities (e.g., multispectral, lidar) often leads to performance degradation due to data scarcity.
• This data limitation can impede the adoption and market entry of new sensor technologies.

Purpose of the Study:

• To present and validate a knowledge distillation approach for improving deep learning model performance during sensor modality transitions.
• To accelerate the adaptation of models to new sensor data by leveraging knowledge from pre-trained teacher networks.
• To investigate extensions of knowledge distillation for enhanced performance in multimodal learning scenarios.

Main Methods:

• Implemented knowledge distillation to transfer knowledge from a teacher network (trained on original modality) to a student network (trained on new modality).
• Validated the approach on multimodal versions of MNIST and CIFAR-10 datasets.
• Explored extensions including annealing of the hyperparameter α and selective knowledge distillation.

Main Results:

• Knowledge distillation significantly improved student network performance on new modalities, especially with limited data (e.g., 10 images).
• On MNIST, test set accuracy increased from 0.37 (baseline) to 0.77 with teacher supervision.
• An annealing scheme for hyperparameter α yielded the best results on CIFAR-10, improving test set accuracy by 6%.

Conclusions:

• Knowledge distillation is an effective method to overcome data scarcity challenges when transitioning to new sensor modalities.
• The proposed approach accelerates the development and deployment of deep learning models for novel sensor technologies.
• The method shows promise for real-world applications, such as skin lesion classification, demonstrating its practical utility.