Diagnostic Agreement Between a General-Purpose AI Model and Retinal Specialists in Color Fundus Photography-A Pilot...

Area of Science:

• Ophthalmology
• Medical Artificial Intelligence
• Computational Imaging

Background:

• Artificial intelligence (AI) shows promise in disease-specific retinal screening.
• Reliability of general-purpose AI in diverse clinical settings is uncertain.
• This study evaluates a multimodal AI model against human specialists for retinal image interpretation.

Purpose of the Study:

• To compare the diagnostic agreement of a general-purpose AI model with retinal specialists.
• To assess the reliability of AI in interpreting color fundus photographs (CFPs) in a clinical context.
• To evaluate AI performance in open-ended retinal image assessment.

Main Methods:

• Pilot retrospective cross-sectional study.
• 66 color fundus photographs (CFPs) evaluated by a masked retinal specialist and Google Gemini 2.5 Flash AI.
• Comparison against an unblinded treating specialist with full clinical information.
• Agreement assessed using weighted percent agreement and Gwet's AC2.

Main Results:

• Substantial agreement between two human specialists (AC2 = 0.67).
• Low agreement between the AI model and the reference specialist (AC2 = -0.58).
• Limited reliability in direct comparison between masked specialist and AI (AC2 = -0.38).

Conclusions:

• The evaluated AI model showed limited agreement with a context-informed specialist.
• Findings suggest cautious interpretation of consumer-facing AI for retinal image assessment.
• Further validation in larger, multicenter studies is warranted.