SCMamba-SAM: transferring knowledge from natural images to medical tasks

The Segment Anything Model (SAM) has drawn significant attention because of its powerful performance in generating high-quality objectmasks. Despite its superior performance in handling natural images, SAM exhibits significant performance degradation when applied to medical images due to the substantial domain gap between natural and medical images. In addition, the specific characteristics of medical images, such as strong heterogeneity, low contrast, and wide individual variation, further raise dificulties in segmenting medical images.

To bridge the semantic gap between natural and medical images, we propose an adaption framework, named SCMamba-SAM. It employs a foundational image encoder constructed by integrating the pre-trained encoder of SAM and Low-Rank Adaption technique. The Encoding Mamba Adapter adopts the spatial- and channel-Mamba blocks to capture task-related global features. Additionally, an enhanced mask decoder with deep supervision has been developed to eliminate the requirement for a prompt encoder, thereby transforming SAM into a fully automated and highly accurate image segmentation model.

Extensive experiments were performed on six public lesion segmentation datasets in different imaging modalities and body parts. The proposed SCMamba-SAM not only achieved performance superior to that of SOTA methods, quantitatively and qualitatively, but also exhibited high generalization.

SCMamba-SAM holds promise as a useful tool for clinicians to diagnose and quantify lesions, providing important information for treatment decisions and surgery planning.