FreeKD+: A Frequency Knowledge Distillation Framework for Dense Prediction

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Knowledge distillation (KD) commonly uses spatial imitation losses for dense prediction tasks.
• Spatial downsampling in feature pyramids causes distortions, hindering accurate feature imitation and degrading student model performance.

Purpose of the Study:

• To address accuracy degradation in dense prediction tasks caused by spatial distortions during knowledge distillation.
• To propose FreeKD, a novel frequency knowledge distillation framework for optimal localization and extent of distillation.

Main Methods:

• Frequency Prompts are integrated into the teacher model to capture semantic frequency context.
• A pixel-wise frequency mask is generated to localize features across different frequency bands.
• A position-aware relational frequency loss enhances the student model's spatial understanding.
• A frequency-decoupled strategy balances alignment for high- and low-frequency signals.
• Frequency masks are refined by reconstructing regions of interest based on student knowledge.

Main Results:

• FreeKD framework demonstrates effectiveness in improving dense prediction tasks.
• Experimental results on COCO, VisDrone, Cityscapes, ADE20K, and COCO-C datasets validate the proposed approach.
• The frequency-decoupled strategy and mask refinement optimize the distillation process.

Conclusions:

• FreeKD effectively overcomes limitations of spatial imitation losses in knowledge distillation for dense prediction.
• The proposed frequency knowledge distillation approach enhances student model performance by analyzing corrupted feature maps in the frequency domain.
• FreeKD offers a robust and effective method for improving accuracy in various computer vision applications.