Distilling Object Detectors via Monte Carlo Dropout.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Knowledge distillation (KD) is crucial for compressing object detection models.
• Teacher model knowledge can be unreliable due to data noise and training randomness, termed knowledge uncertainty.
• Existing KD methods often overlook this uncertainty, limiting student model performance.

Purpose of the Study:

• To introduce a novel strategy, Uncertainty-Driven Knowledge Extraction and Transfer (UET), to explicitly incorporate knowledge uncertainty in KD.
• To enhance the student model's ability to capture latent 'dark knowledge' by accounting for teacher uncertainty.

Main Methods:

• Employed Monte Carlo dropout to estimate teacher model uncertainty.
• Utilized information theory to combine uncertainty with deterministic knowledge for improved learning.
• Developed UET as a plug-and-play module for seamless integration with existing KD techniques.

Main Results:

• UET achieved state-of-the-art performance in object detection model compression.
• A ResNet50-based GFL detector using UET attained 44.1% mAP on the COCO dataset.
• This represents a 3.9% improvement over baseline performance, demonstrating UET's effectiveness.

Conclusions:

• UET effectively addresses knowledge uncertainty in KD for object detection.
• The proposed method enhances student model learning by leveraging both precision and diversity in knowledge transfer.
• UET offers a significant advancement in model compression techniques, showing broad applicability across various architectures and strategies.