In-Context Learning for Data-Efficient Diabetic Retinopathy Detection via Multimodal Foundation Models.

Area of Science:

• Artificial Intelligence in Medicine
• Medical Image Analysis
• Ophthalmology

Background:

• Multimodal foundation models offer rapid adaptation to new tasks via in-context learning (ICL).
• Domain-specific models require extensive fine-tuning for specialized tasks like diabetic retinopathy (DR) detection.

Purpose of the Study:

• To evaluate if ICL using Google Gemini 1.5 Pro can match the diagnostic performance of a domain-specific model (RETFound) for DR detection.
• To compare the efficacy of a general multimodal model with ICL against a specialized, fine-tuned model.

Main Methods:

• A retrospective dataset of 516 color fundus photographs (CFPs) was used for binary DR classification.
• Google Gemini 1.5 Pro was tested with zero-shot and few-shot ICL.
• RETFound was fine-tuned for DR detection.
• Performance was evaluated using accuracy, F1 score, and calibration error with 10-fold cross-validation.

Main Results:

• Gemini 1.5 Pro with ICL achieved an accuracy of 0.841 and F1 score of 0.876.
• RETFound achieved an accuracy of 0.849 and F1 score of 0.883.
• While accuracy and F1 scores were comparable, RETFound demonstrated superior calibration (P = 0.004).

Conclusions:

• In-context learning with Gemini 1.5 Pro shows comparable performance to domain-specific models for diabetic retinopathy detection.
• This highlights the potential for adaptable foundation models in future medical AI systems.
• Future systems may rely less on bespoke solutions and more on versatile, pre-trained models.