Multi-Modal AI for Multi-Label Retinal Disease Prediction Using OCT and Fundus Images: A Hybrid Approach.

Area of Science:

• Ophthalmology and Artificial Intelligence
• Medical Diagnostics
• Computer Vision

Background:

• Ocular diseases significantly impact vision and quality of life.
• Current diagnostic methods are time-consuming and rely on expert interpretation.
• Existing AI systems often focus solely on medical imaging.

Purpose of the Study:

• To develop and evaluate VisionTrack, a multi-modal AI system for predicting multiple retinal diseases.
• To integrate image-based, clinical risk factor, and text-based data for comprehensive retinal health assessment.
• To improve accuracy and efficiency in diagnosing conditions such as Diabetic Retinopathy (DR), Age-related Macular Degeneration (AMD), and others.

Main Methods:

• Utilized a hybrid AI framework integrating Convolutional Neural Networks (CNNs), Graph Neural Networks (GNNs), and Large Language Models (LLMs).
• Employed CNNs for extracting features from retinal images (OCT and fundus).
• Applied GNNs to model relationships within clinical risk factors and LLMs to process patient medical reports.

Main Results:

• VisionTrack demonstrated strong multi-label disease prediction performance on RetinalOCT and RFMID datasets.
• Achieved high accuracy (e.g., 0.980 on RetinalOCT, 0.989 on RFMID) and F1-scores across various retinal conditions.
• Confirmed robustness, reliability, and generalization capabilities across diverse imaging modalities.

Conclusions:

• The multi-modal AI system VisionTrack offers a comprehensive approach to retinal disease prediction.
• This hybrid system shows significant potential for early detection, risk assessment, and personalized ophthalmic care.
• The integration of diverse data sources enhances diagnostic accuracy and clinical utility.