RETHINKING INTERMEDIATE LAYERS DESIGN IN KNOWLEDGE DISTILLATION FOR KIDNEY AND LIVER TUMOR SEGMENTATION

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Knowledge distillation (KD) is successful but not optimized for medical imaging segmentation.
• Existing KD methods often fail to select optimal knowledge sources and transfer points.
• This can lead to training bias in student models, hindering KD effectiveness.

Purpose of the Study:

• To introduce Hierarchical Layer-selective Feedback Distillation (HLFD) for improved medical image segmentation.
• To address limitations in current KD approaches for tasks like kidney and liver tumor segmentation.
• To develop a method for more effective knowledge transfer from teacher to student models.

Main Methods:

• Proposed HLFD method for strategic knowledge distillation.
• Distills knowledge from middle to earlier layers and final to intermediate layers.
• Utilizes both feature and pixel-level knowledge transfer.
• Focuses on learning higher-quality representations in earlier layers.

Main Results:

• HLFD significantly outperforms existing KD methods in quantitative evaluations.
• Achieved over 10% improvement in kidney segmentation compared to a student model without KD.
• Demonstrated enhanced focus on tumor-specific features.
• Student models trained with HLFD excel at suppressing irrelevant information.

Conclusions:

• HLFD offers a novel pathway for efficient and accurate medical diagnostic tools.
• The method enables the development of robust and compact student models for segmentation.
• HLFD effectively addresses the challenges of knowledge transfer in medical imaging KD.