Knowledge Distillation in Object Detection: A Survey from CNN to Transformer.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Deep learning object detection models offer high accuracy but are computationally intensive.
• Deployment on resource-constrained devices (e.g., mobile phones) is challenging due to model size and complexity.
• Knowledge Distillation (KD) is a key technique for model compression, creating efficient student models from complex teacher models.

Purpose of the Study:

• To provide a comprehensive review of Knowledge Distillation (KD) applied to object detection models.
• To analyze existing KD techniques, their strengths, weaknesses, and potential future research avenues.
• To explore extended applications of KD in object detection and related computer vision domains.

Main Methods:

• Systematic review of recent research on KD-based object detection.
• Analysis of various distillation algorithms and their effectiveness.
• Examination of KD applications for lightweight models, incremental learning, and small object detection.

Main Results:

• KD effectively reduces the size and computational cost of object detection models while preserving accuracy.
• Various KD techniques offer different trade-offs between compression and performance.
• KD shows promise in enhancing performance for specific challenges like small object detection and incremental learning.

Conclusions:

• Knowledge Distillation is a vital strategy for deploying advanced object detection models on edge devices.
• Further research is needed to optimize KD algorithms and explore novel applications.
• KD's principles extend beyond object detection, impacting various computer vision tasks.