Related Experiment Videos
SUDA: Simultaneous unsupervised knowledge distillation and adaptation of foundation models for efficient pathological
Lanfeng Zhong1, Kun Qian2, Weiren Zhao3
1School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
Medical Image Analysis
|June 27, 2026
Summary
This study introduces Simultaneous Unsupervised Knowledge Distillation and Adaptation (SUDA) to create smaller, efficient pathology foundation models. SUDA effectively compresses large models while maintaining high performance on new datasets without needing manual labels.
Area of Science:
- Artificial Intelligence
- Medical Image Analysis
- Computational Pathology
Background:
- Pathology foundation models offer advanced image analysis but suffer from large size and domain shift issues.
- Existing knowledge distillation methods require labeled data and do not address domain shifts effectively.
Purpose of the Study:
- To develop a method for compressing large pathology foundation models into lightweight versions.
- To improve model performance on downstream tasks despite domain shifts and reduce computational costs.
Main Methods:
- Simultaneous Unsupervised Knowledge Distillation and Adaptation (SUDA) integrates knowledge distillation with self-supervised learning.
- SUDA employs Dual Instance Discrimination Distillation (DI2D) and Masked Consistency Modeling (MCM) for unsupervised adaptation and compression.
- The approach enables adaptation to downstream pathology datasets without requiring human annotations.
Main Results:
- SUDA outperformed existing knowledge distillation and self-supervised learning methods on multiple pathology datasets.
- The compressed student models achieved performance comparable to or exceeding the larger teacher models.
- SUDA reduced model parameters by 0.018×, significantly lowering computational costs.
Conclusions:
- SUDA offers an effective solution for creating high-performing, lightweight pathology foundation models.
- The method addresses key limitations of model size, computational cost, and domain adaptation in pathological image analysis.