Two-stage optimization based on heterogeneous branch fusion for knowledge distillation.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Knowledge distillation effectively transfers knowledge from teacher to student models.
• Sole reliance on fixed teacher knowledge limits student model generalization.
• Existing methods lack dynamic knowledge supplementation for improved adaptability.

Purpose of the Study:

• To propose a two-stage optimization method based on heterogeneous branch fusion for knowledge distillation (THFKD).
• To enhance student model generalization ability by providing stage-appropriate knowledge.
• To improve classification accuracy and adaptability on standard and long-tail datasets.

Main Methods:

• Implemented a two-stage optimization strategy with heterogeneous branch fusion.
• Utilized a pre-trained teacher model for stable static knowledge transfer.
• Employed a progressive feature fusion module for dynamic knowledge generation in the student model.
• Applied a ramp-up loss weight in the first stage and consistent weights in the second.

Main Results:

• THFKD demonstrated improved classification accuracy and generalization ability on CIFAR-100, Tiny-ImageNet, and CIFAR100-LT datasets.
• Achieved a 1.52% accuracy improvement on CIFAR-100 using ResNet110-ResNet32, reaching 75.41%.
• The method effectively balances static and dynamic knowledge for enhanced student performance.

Conclusions:

• THFKD offers a promising approach to knowledge distillation by leveraging heterogeneous branch fusion and staged optimization.
• The proposed method enhances student model performance and generalization, particularly on challenging datasets.
• Further validation with statistical significance tests is recommended to confirm the observed improvements.