A Foundation Language-Image Model of the Retina (FLAIR): encoding expert knowledge in text supervision.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Foundation vision-language models show promise in computer vision but struggle with medical imaging due to domain shifts and complex expert knowledge.
• Existing models exhibit limitations in medical imaging tasks compared to other domains.

Purpose of the Study:

• To introduce FLAIR, a pre-trained vision-language model specifically designed for universal retinal fundus image understanding.
• To address the challenges of domain shift and incorporate expert knowledge into medical imaging foundation models.

Main Methods:

• Compiled 38 open-access fundus imaging datasets (288,307 images, 101 conditions).
• Integrated expert domain knowledge via descriptive textual prompts during pre-training and zero-shot inference.
• Utilized clinical literature and community standards to create detailed textual descriptions of pathologies and their relationships.

Main Results:

• FLAIR demonstrates strong generalization capabilities, even with domain shifts and unseen categories.
• When fine-tuned with a linear probe, FLAIR surpasses fully-trained, dataset-specific models, particularly in few-shot learning.
• FLAIR significantly outperforms larger generalist image-language models and existing retina-specific self-supervised networks.

Conclusions:

• Integrating expert knowledge is crucial for developing effective foundation models in medical imaging.
• FLAIR offers a powerful solution for universal retinal fundus image understanding, highlighting the limitations of generalist models.
• The pre-trained FLAIR model shows significant potential for advancing AI in ophthalmology and medical diagnostics.