Cosine similarity-guided knowledge distillation for robust object detectors.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Knowledge Distillation (KD) is effective for image classification but faces challenges in object detection due to its complexity.
• Traditional KD methods for object detection often rely on Mean Squared Error (MSE) loss and have limited feature representation.
• Object detection models require robust and lightweight designs for various applications.

Purpose of the Study:

• To develop a Cosine Similarity-Based Knowledge Distillation (CSKD) method for robust and lightweight object detectors.
• To address the limitations of traditional KD techniques in object detection.
• To improve knowledge transfer from teacher to student models in object detection.

Main Methods:

• CSKD combines cosine similarity guidance with MSE loss for effective knowledge transfer.
• The method distills both intermediate features and prediction outputs using an assistant prediction branch.
• It enables student models to better mimic teacher model behavior without additional feature enhancement layers.

Main Results:

• CSKD demonstrates versatility and robustness across multiple object detector architectures (Faster-RCNN, RetinaNet, FCOS, GFL).
• Using ResNet-50 as teacher and ResNet-18 as student, new benchmarks in KD for object detection were achieved.
• Specific mAP scores include 36.6 for Faster-RCNN, 35.2 for RetinaNet, 35.9 for FCOS, and 38.9 for GFL.

Conclusions:

• CSKD effectively advances the state-of-the-art in knowledge distillation for object detection.
• The proposed method offers a compelling solution to challenges in traditional KD for object detection.
• CSKD provides a robust and versatile approach for developing lightweight and high-performing object detectors.