TA-MedSAM: Text-augmented improved MedSAM for pulmonary lesion segmentation.

Area of Science:

• Medical Image Analysis
• Artificial Intelligence
• Computer Vision

Background:

• Accurate lung lesion segmentation is vital for clinical diagnosis but limited by traditional unimodal visual data methods.
• Existing medical image segmentation models struggle with lesions exhibiting low contrast, blurred boundaries, complex morphology, and small size.

Purpose of the Study:

• To develop a novel multimodal approach, TA-MedSAM, for enhanced pulmonary lesion segmentation.
• To improve segmentation accuracy by integrating visual and textual information through a vision-language fusion paradigm.

Main Methods:

• Introduced a lightweight Medical Segment Anything Model (MedSAM) image encoder and a pre-trained ClinicalBERT text encoder.
• Developed a Reconstruction Text Module for lesion-centric textual cue focus and a Multimodal Feature Fusion Module with attention mechanisms.
• Implemented a feature alignment coordination mechanism, Dynamic Perception Learning Mechanism, and Multi-scale Feature Fusion Module with a Multi-task Loss Function.

Main Results:

• TA-MedSAM significantly improves segmentation accuracy for challenging pulmonary lesions.
• The model demonstrates reduced parameters and computational costs, enhancing inference speed.
• Comparative experiments show TA-MedSAM outperforms state-of-the-art unimodal and multimodal methods on multiple datasets.

Conclusions:

• TA-MedSAM offers a superior approach to lung lesion segmentation by effectively leveraging multimodal information.
• The proposed components and fusion strategies contribute to enhanced segmentation performance, particularly for complex cases.
• This method holds promise for advancing clinical diagnosis and treatment planning for pulmonary conditions.