Knowledge Fusion Distillation: Improving Distillation with Multi-scale Attention Mechanisms

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Deep learning models achieve high performance but require significant computational resources, limiting their use on edge devices.
• Knowledge distillation (KD) compresses models by transferring knowledge from large teacher models to smaller student models.
• Self-distillation (SD) avoids pre-trained teachers but often neglects early model features and relies on suboptimal deep layer guidance.

Purpose of the Study:

• To address limitations in existing self-distillation methods regarding the utilization of early model features.
• To propose a novel self-distillation technique that effectively leverages feature fusion for improved model compression and performance.
• To enhance the applicability of deep learning models on resource-constrained edge devices.

Main Methods:

• Introduced a selective feature fusion module to better integrate features from different network layers.
• Developed a new self-distillation method named knowledge fusion distillation (KFD).
• Conducted extensive experiments on three diverse datasets to validate the proposed method.

Main Results:

• Knowledge fusion distillation demonstrated comparable performance to state-of-the-art distillation techniques.
• The proposed method shows potential for further performance enhancement when fused features are integrated into the network.
• The approach effectively utilizes early-stage features, overcoming a key limitation of previous self-distillation methods.

Conclusions:

• Knowledge fusion distillation offers an effective approach to model compression and performance enhancement for deep learning.
• The method provides a viable solution for deploying complex deep learning models on edge devices with limited resources.
• Selective feature fusion is a promising strategy for advancing self-distillation techniques in deep learning.