Medical SAM-Clip Grafting for brain tumor segmentation

Brain Tumor Segmentation (BTS) is crucial for accurate diagnosis and treatment planning, but existing CNN and Transformer-based methods often struggle with feature fusion and limited training data. While recent large-scale vision models like Segment Anything Model (SAM) and CLIP offer potential, SAM is trained on natural images, lacking medical domain knowledge, and its decoder struggles with accurate tumor segmentation. To address these challenges, we propose the Medical SAM-Clip Grafting Network (MSCG), which introduces a novel SC-grafting module. This module grafts the rich semantic information from CLIP's feature space into the SAM model, enhancing its ability to capture both pixel-level details and high-level semantic context. By effectively combining SAM's visual precision with CLIP's semantic richness, we introduce SC-grafting module to improve the model's robustness and accuracy in brain tumor segmentation, especially in cases with irregular shapes, low contrast, or complex anatomical structures. Extensive experiments on the BraTS dataset demonstrate that MSCG significantly outperforms existing methods, showcasing its effectiveness in delivering more precise tumor segmentation and its potential in advancing medical image analysis.