Expandable Residual Approximation for Knowledge Distillation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Knowledge distillation (KD) transfers knowledge from large teacher models to smaller student models, reducing computational costs.
• A significant challenge in KD is the learning capacity gap between teacher and student models, hindering effective knowledge transfer.

Purpose of the Study:

• To propose a novel knowledge distillation method, Expandable Residual Approximation (ERA), inspired by the Stone-Weierstrass theorem.
• To address the capacity gap in KD by decomposing residual knowledge approximation and integrating teacher weights.

Main Methods:

• ERA decomposes residual knowledge approximation into multiple steps using a multibranched residual network (MBRNet).
• A teacher weight integration (TWI) strategy is employed to mitigate the capacity disparity by reusing teacher head weights.

Main Results:

• ERA achieved a 1.41% improvement in Top-1 accuracy on the ImageNet classification benchmark.
• ERA improved the AP by 1.40 on the MS COCO object detection benchmark.
• The method demonstrated leading performance across various computer vision tasks.

Conclusions:

• ERA effectively bridges the knowledge transfer gap in KD through a divide-and-conquer approach.
• The proposed method offers a promising solution for efficient and effective model compression in computer vision.