Comprehensive Evaluation of ChatGPT's Diagnostic Accuracy on Image-based Ophthalmic Case Interpretations

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Artificial intelligence (AI) shows promise in medical diagnostics.
• Evaluating AI performance requires assessing its accuracy with varying data inputs.

Purpose of the Study:

• To assess the diagnostic and treatment accuracy of Chat Generative Pre-trained Transformer (GPT-4.o) in ophthalmology.
• To compare GPT-4.o performance using full clinical context versus image-only inputs.

Main Methods:

• A cross-sectional study analyzed 261 ophthalmic cases from the EyeRounds repository.
• Cases were evaluated with GPT-4.o using full clinical context and image-only inputs.
• Outputs were compared to expert diagnoses and evaluated for accuracy and component match rates.

Main Results:

• Diagnostic accuracy was significantly higher with full-context input (80.1%) versus image-only input (54.7%).
• Match rates for signs/symptoms, differential diagnoses, and treatment recommendations were consistently higher with full context.
• Pediatric cases showed the highest full-context accuracy, while glaucoma and neuro-ophthalmology had the lowest image-only accuracy.

Conclusions:

• GPT-4.o demonstrates high diagnostic accuracy in ophthalmology when provided with complete clinical information.
• Performance significantly declines with image-only input, indicating limitations in current multimodal AI for raw image interpretation.
• Integrating structured clinical data is crucial for optimizing AI-driven diagnostic support in ophthalmology.