Semantic AutoSAM: Self-Prompting Segment Anything Model for Semantic Segmentation of Medical Images

Segment Anything Model (SAM) is a foundation model that can be prompted with sparse prompts, like boxes or points, and dense prompts such as masks. SAM outputs binary masks based on the given prompts but lacks semantic understanding as it doesn't output the class of the predicted mask. We propose Semantic AutoSAM, a semantic segmentation model that builds upon SAM's binary segmentation. Semantic AutoSAM replaces SAM's manual prompt encoder with a lightweight cross-attention module, enabling it to predict prompt embeddings directly from the image features. This eliminates the need for manual prompting.In our experiments on the FLAIR 2022 dataset (20 CT scans) and a hip ultrasound dataset (4849 2D images), Semantic AutoSAM matches the performance of using groundtruth bounding box prompts for most organs. Our proposed method achieves a Dice score of 0.62 in the FLAIR dataset, and MobileSAM with groundtruth box achieves 0.7. In the hip ultrasound dataset, our approach achieves a Dice score of 0.83, surpassing MobileSAM's slightly lower score of 0.81 despite MobileSAM having access to the groundtruth box for prediction. Notably, our method doesn't require manual prompts at test time.