SUDA: Simultaneous unsupervised knowledge distillation and adaptation of foundation models for efficient pathological...

Pathology foundation models have greatly advanced pathological image analysis due to their generalizable feature representation capabilities after learning from a large dataset. However, they still face two critical limitations: first, their large model size and high computational cost hinder clinical deployment; second, domain shifts between pretraining and downstream datasets degrade performance on target tasks. While existing Knowledge Distillation (KD) methods have shown effectiveness in model compression, they typically rely on fully labeled data and overlook domain shifts between pretraining and downstream tasks. To overcome these limitations, we propose Simultaneous Unsupervised knowledge Distillation and Adaptation (SUDA) that integrates KD with self-supervised learning, enabling effective adaptation and compression of foundation models into lightweight ones with high performance on target domains. Specifically, SUDA introduces Dual Instance Discrimination Distillation (DI2D) that contains teacher-guided instance and relation-wise distillation and student self-guided instance discrimination distillation. We further propose Masked Consistency Modeling (MCM) to improve the student model's feature consistency under masking, enabling better adaptation to downstream pathology datasets without human annotations. Extensive experiments on two patch-level and three WSI-level pathology image datasets demonstrate that our SUDA outperformed existing knowledge distillation and self-supervised learning methods under different student and teacher structures. What is more, the student after SUDA matches or even surpasses the performance of the teacher model with only 0.018× parameters, significantly reducing the computational cost.