Heterogeneous Feature Knowledge Distillation based on Enhanced Feature Projector Correlation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge Distillation (KD) typically assumes homogeneous teacher-student architectures, limiting its application to diverse models.
• Heterogeneous architectures present challenges in feature alignment due to differences in structure and representation.

Purpose of the Study:

• To propose a novel heterogeneous feature knowledge distillation method for improved model performance.
• To address the limitations of existing KD methods in aligning dissimilar model architectures.

Main Methods:

• Developed a heterogeneous feature knowledge distillation approach using enhanced feature projector correlation.
• Implemented a cross-space fusion mechanism to maintain semantic relevance during feature projection.
• Introduced multi-level feature distillation loss and a diffusion model-based denoising mechanism.

Main Results:

• Validated the method's effectiveness across diverse architectures (CNNs, Transformers, MLPs) on CIFAR-100 and ImageNet datasets.
• Demonstrated successful application in semantic segmentation tasks on the Cityscapes dataset.
• Achieved enhanced class-wise discriminability in student models.

Conclusions:

• The proposed heterogeneous feature knowledge distillation effectively bridges performance gaps between diverse model architectures.
• The method offers a robust solution for knowledge transfer in scenarios with heterogeneous teacher-student models.
• This work advances KD techniques for broader applicability in complex deep learning systems.