ECP-KD: Efficient Computational Pathology Heterogeneous Model Fusion Using Knowledge Distillation.

Area of Science:

• Computational pathology
• Artificial intelligence in medicine
• Machine learning for healthcare

Background:

• Whole Slide Images (WSIs) feature extraction is crucial for computational pathology but demands significant computational resources.
• Knowledge distillation transfers knowledge from large teacher models to smaller student models for efficient deployment.
• Distributional gaps between heterogeneous teacher models (different structures/modalities) pose challenges for effective knowledge distillation.

Purpose of the Study:

• To propose a novel knowledge distillation method, Efficient Computational Pathology Heterogeneous Model Knowledge Distillation (ECP-KD), to bridge distributional gaps.
• To enable effective knowledge transfer from diverse teacher models to student models in computational pathology.
• To improve the efficiency and accuracy of computational pathology models for clinical applications.

Main Methods:

• Developed ECP-KD utilizing structure adapter and MIL adapter layers to bridge the distributional gap.
• Incorporated cross-attention mechanisms for fusing knowledge from multiple pre-trained models and modalities.
• Applied the method to multi-instance learning tasks, including smart WSIs analysis.

Main Results:

• ECP-KD effectively handles network mismatch problems between teacher and student models.
• Experimental results on The Cancer Genome Atlas demonstrated improved student model performance in survival prediction.
• Achieved up to 72x speedup and 33x parameter reduction compared to larger ViT teacher models, maintaining state-of-the-art accuracy.

Conclusions:

• ECP-KD enables efficient and accurate computational pathology by overcoming challenges of heterogeneous model distillation.
• The method is suitable for resource-constrained clinical settings, enhancing model performance without sacrificing computational efficiency.
• Facilitates the deployment of advanced AI models in clinical practice for improved patient outcomes.