Harnessing Text Insights With Visual Alignment for Medical Image Segmentation.

Area of Science:

• Computer Vision
• Medical Image Analysis
• Artificial Intelligence

Background:

• Vision-language models (VLMs) and language models (LMs) show promise for integrating text and image data in vision tasks.
• Current text-enhanced medical image segmentation methods struggle with semantic shifts and feature misalignment between vision and text components.

Purpose of the Study:

• To propose TeViA, a novel approach for seamless integration of diverse vision and text models in medical image segmentation.
• To address semantic shifts and improve text-vision alignment during segmentation tasks.

Main Methods:

• TeViA employs a segmentation-specific text-to-vision alignment strategy.
• It uses foreground visual representations to supervise projection layers, refining textual features for segmentation.
• A historic visual prototype, updated via momentum, enhances instance representation and refines textual features.

Main Results:

• TeViA demonstrated superior performance across five public medical image segmentation datasets.
• The method achieved over 6% Dice improvement compared to vision-only approaches.
• The approach ensures both information gain and semantic consistency in text-enhanced segmentation.

Conclusions:

• TeViA offers an effective solution for text-enhanced medical image segmentation by improving text-vision alignment.
• The method's flexibility allows integration with various pre-trained models, regardless of their initial relationships.
• TeViA significantly advances the state-of-the-art in medical image segmentation accuracy.