ECP-KD: Efficient Computational Pathology Heterogeneous Model Fusion Using Knowledge Distillation

Extracting features from Whole Slide Images (WSIs) using pre-trained models is essential for computational pathology. However, this process usually requires substantial computational resources. Knowledge distillation aims to effectively transfer knowledge from complex, pre-trained teacher models to smaller, more efficient student models, which can be deployed in clinical scene with limited computational resources. Unfortunately, this process becomes especially challenging when teacher models come from different structures or modalities, leading to distributional gaps. To address these challenges, we propose a novel distillation method, named Efficient Computational Pathology Heterogeneous Model Knowledge Distillation (ECP-KD), which utilizes structure adapter layer and MIL adapter layer to bridge the distributional gap between teacher and student models. ECP-KD can effectively handle network mismatch problems with different structures in distillation and typical multi-instance learning tasks such as smart WSIs analysis tasks. We also incorporate cross-attention mechanisms for the fusion of multiple pre-trained models, enabling dynamic and scalable integration of various modalities. Experimental results on The Cancer Genome Atlas demonstrate that the proposed ECP-KD improves the performance of student models in survival prediction tasks, offering a speedup of up to 72x and reducing the number of parameters by up to 33x compared to larger ViT teacher models, with the state-of-the-art accuracy.Clinical relevance- This method enables more efficient and accurate in computational pathology tasks, making it applicable to resource-constrained clinical settings by improving model performance without compromising computational efficiency.