Insight: A Multi-Modal Diagnostic Pipeline using LLMs for Ocular Surface Disease Diagnosis

Area of Science:

• Ophthalmology and Optometry
• Artificial Intelligence in Medicine
• Medical Diagnostics

Background:

• Accurate diagnosis of ocular surface diseases is crucial but challenged by imprecise human assessments and limitations of current AI models.
• Existing machine-based methods often use closed-set classification, restricting diagnoses and lacking clinical variable reasoning.
• Integrating diverse data like meibography images and clinical metadata is key for precise diagnosis.

Purpose of the Study:

• To develop an innovative multi-modal diagnostic pipeline (MDPipe) for ocular surface disease diagnosis.
• To leverage large language models (LLMs) for enhanced diagnostic accuracy and clinical reasoning.
• To overcome the limitations of traditional and current AI diagnostic approaches.

Main Methods:

• Developed MDPipe, a pipeline integrating meibography image interpretation and clinical metadata using LLMs.
• Employed a visual translator to convert meibography images into quantifiable morphology data.
• Utilized an LLM-based summarizer for contextualizing integrated data and generating clinical report summaries, refined with clinician input.

Main Results:

• MDPipe demonstrated superior performance across diverse ocular surface disease diagnosis benchmarks compared to existing standards, including GPT-4.
• The pipeline successfully integrated visual and clinical data for nuanced diagnostic insights.
• MDPipe provided clinically sound rationales supporting its diagnoses.

Conclusions:

• MDPipe represents a significant advancement in the AI-assisted diagnosis of ocular surface diseases.
• The multi-modal approach effectively combines imaging, metadata, and LLM reasoning for improved diagnostic outcomes.
• This pipeline offers a more comprehensive and interpretable approach to diagnosing complex eye conditions.