Feature distance-weighted adaptive decoupled knowledge distillation for medical image segmentation.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Medical Imaging

Background:

• Deep learning models for medical image segmentation often require significant computational resources, hindering deployment on resource-constrained embedded devices.
• Knowledge distillation offers a way to transfer knowledge from large, high-performance models to smaller, more efficient ones.

Purpose of the Study:

• To apply decoupled knowledge distillation (DKD) to medical image segmentation, enabling the deployment of lightweight student networks on embedded devices.
• To improve the efficiency and effectiveness of knowledge transfer in medical image segmentation models.

Main Methods:

• Decoupled the distillation loss into pixel-wise target class knowledge distillation (PTCKD) and pixel-wise non-target class knowledge distillation (PNCKD).
• Proposed a novel feature distance-weighted adaptive decoupled knowledge distillation (FDWA-DKD) method to address limitations of fixed weights in PTCKD.
• Introduced a feature distance weighting (FDW) module for instance-level adaptive weights and a class-wise feature probability distribution loss.

Main Results:

• FDWA-DKD achieved optimal Dice scores on Synapse and FLARE22 datasets.
• The proposed method demonstrated performance comparable to, and in some cases exceeding, the teacher network.
• Ablation studies confirmed the effectiveness of individual modules within the FDWA-DKD framework.

Conclusions:

• FDWA-DKD overcomes traditional distillation constraints by providing instance-level adaptive weights for PTCKD.
• Quantifying student-teacher feature disparity and minimizing class-wise feature probability distribution loss leads to superior performance.
• The method offers a viable solution for deploying high-performance medical image segmentation models on embedded systems.