Leveraging Multi-Text Joint Prompts in SAM for Robust Medical Image Segmentation.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• The Segment Anything Model (SAM) shows promise for medical image segmentation.
• Text prompts offer a more efficient alternative to geometric prompts for SAM in medicine.
• Existing methods for text-prompted SAM in medicine are underexplored.

Purpose of the Study:

• To propose a novel SAM-based framework utilizing vision-language models for text-prompted medical image segmentation.
• To address the challenges of semantic ambiguity and information redundancy in clinical text descriptions.
• To enhance the performance and usability of SAM for medical applications through optimized text prompts.

Main Methods:

• Integrated a pre-trained vision-language model (e.g., CLIP) to generate referring prompts for SAM.
• Developed a text decomposition-recomposition strategy to parse and recombine clinical narratives into effective semantic units.
• Employed a cross-attention module to refine feature focus based on optimized text descriptions.

Main Results:

• The proposed SAM-based framework demonstrated improved performance compared to native SAM with geometric prompts.
• The text decomposition-recomposition strategy effectively handled complex medical text descriptions.
• Experiments on multiple datasets validated the enhanced usability and accuracy of the method.

Conclusions:

• The developed framework offers a viable and effective approach for text-prompted medical image segmentation using SAM.
• Optimizing text prompts through decomposition and recomposition is crucial for leveraging vision-language models with SAM in clinical settings.
• This method presents a significant advancement in the application of foundation models for medical image analysis.