RFD: A Reducing Feature Discrepancy method for unsupervised cross-modality SAM adaptation

Vision foundation models, such as Segment Anything Model (SAM), have achieved great success in medical image segmentation tasks under full supervision, but require accurate masks. However, SAM suffers from significant performance degradation when applied to unsupervised cross-modality medical image segmentation. To maintain satisfactory performance and reduce the masking cost, we propose a Reducing Feature Discrepancy (RFD) method for SAM in cross-modality medical image segmentation, including Reweighting Unbalanced Feature Adaptation (RUFA) and Structural Prototype-based Contrastive Learning (SPCL). Specifically, we integrate medical adapters into the SAM encoder to improve its feature capture capabilities. SPCL introduces the structural prototypes with the structural information distance instead of simple node embeddings, improving feature discrimination in latent spaces and achieving accurate pixel-prototype transport. We also propose Inexact Prototype-based Pixel Transport (IPPT) in SPCL for the dynamic selection and allocation of prototypes, leading to efficient and robust online clustering. Furthermore, RUFA distinguishes image samples utilizing features from the SAM latent space, contingent upon their similarity or dissimilarity, thereby promoting inter-class discrimination and intra-class compactness with small uncertainty. Our experimental results demonstrate a significant improvement over recent state-of-the-art unsupervised domain adaptation (UDA) works and SAM-based methods on four public datasets, validating its effectiveness and generalization.