Text-Assisted Vision Model for Medical Image Segmentation.

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Computer Vision

Background:

• Accurate medical image segmentation is crucial for automated diagnosis and treatment planning.
• Deep learning models primarily rely on image data, often overlooking valuable information in text reports.
• Existing attention mechanisms struggle with cross-modal alignment, limiting performance in multi-modal scenarios.

Purpose of the Study:

• To develop a novel text-assisted vision (TAV) model for improved medical image segmentation.
• To introduce a triguided attention module (TGAM) for effective cross-modal feature learning.
• To enhance segmentation precision by leveraging both visual and textual data.

Main Methods:

• Proposed a text-assisted vision (TAV) model incorporating a novel triguided attention module (TGAM).
• TGAM computes visual-visual, language-language, and language-visual attention for feature correlation.
• An attention gate (AG) was used to modulate TGAM's influence, preventing information overflow.

Main Results:

• The TAV model achieved state-of-the-art performance on two medical image segmentation datasets.
• TAV demonstrated performance improvements of 2-7% compared to existing models.
• Extensive experiments validated the effectiveness of individual components within the TAV model.

Conclusions:

• The TAV model represents a significant advancement in multi-modal medical image segmentation.
• Integrating text reports via TGAM substantially enhances segmentation accuracy.
• The proposed approach offers a promising direction for leveraging multi-modal data in medical AI.