Resolution-Aware Knowledge Distillation for Efficient Inference.

Area of Science:

• Computer Vision
• Deep Learning
• Machine Learning

Background:

• Deep networks require significant computation, hindering practical application.
• Current acceleration methods focus on network structure or parameter compression.
• Knowledge distillation is used to maintain performance during model compression.

Purpose of the Study:

• To accelerate deep learning models by using low-resolution (LR) images as input.
• To address performance degradation and feature learning challenges with LR inputs.
• To propose a novel resolution-aware knowledge distillation (RKD) framework.

Main Methods:

• Proposed a resolution-aware knowledge distillation (RKD) framework with HR teacher and LR student networks.
• Introduced a discriminator and adversarial learning to minimize cross-resolution variations.
• Designed a cross-resolution knowledge distillation (CRKD) loss including resolution-aware distillation, pair-wise constraint, and MMD loss.

Main Results:

• RKD framework demonstrated superior performance compared to traditional knowledge distillation methods.
• Achieved better performance with reduced computational complexity across various tasks.
• CRKD surpassed state-of-the-art methods in cross-resolution knowledge transfer, especially with large resolution differences.

Conclusions:

• The proposed RKD framework effectively accelerates deep models using LR inputs while preserving performance.
• RKD successfully narrows cross-resolution variations, enabling robust feature learning.
• This approach offers a promising direction for efficient deep learning deployment in resource-constrained environments.