Language-assisted multimodal convolutional transformer pipeline for retinal lesions segmentation.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Imaging

Background:

• Retinal lesion segmentation is crucial for diagnosing eye diseases.
• Current deep learning models often lack clinical relevance and require extensive pixel-level annotations.
• These limitations hinder accurate and efficient retinal disease analysis.

Purpose of the Study:

• To develop a novel language-assisted multimodal convolutional transformer pipeline for retinal lesion segmentation.
• To overcome the limitations of existing models by aligning image and text features.
• To enable robust lesion extraction without pixel-level ground truth annotations.

Main Methods:

• A multimodal convolutional transformer pipeline was designed to align retinal scan image features with text features from clinical prompts.
• A novel loss function was employed for one-time training to establish feature alignment.
• The model was trained to infer learning from text prompts, eliminating the need for dataset-specific pixel-level annotations.

Main Results:

• The proposed network demonstrated robust retinal lesion extraction across six public datasets.
• The model achieved up to 7.77% improvement in intersection-over-union compared to state-of-the-art methods.
• The language-assisted approach proved effective in adapting to new datasets without retraining.

Conclusions:

• The developed language-assisted multimodal pipeline offers a significant advancement in retinal lesion segmentation.
• This approach enhances clinical relevance and reduces the dependency on laborious pixel-level annotations.
• The model shows strong potential for improving the diagnosis and management of retinal diseases.