PAIR: Evaluating the Limits of Agreement Among Non-Retinal Specialist Using PathFinder Artificial Intelligence Tool...

To evaluate the diagnostic and referral agreement between non-retina specialists (NRS) using the PathFinder artificial intelligence (AI) assistant and fellowship-trained retina specialists (RS - gold standard) in interpreting macular optical coherence tomography (OCT) scans.

This cross-sectional study included 202 consecutive patients undergoing OCT on the CIRRUS platform with the PathFinder AI module. Three RS independently graded all scans without clinical data, while three NRS interpreted the same scans using PathFinder assistance and full clinical information. The gold standard for both diagnosis and referral was defined by agreement of at least two of the three RS. The NRS recorded diagnostic confidence and time to AI-assisted decision. Agreement was assessed with Cohen's and Fleiss' κ and sensitivity, and specificity were computed for four major pathologies.

Among 202 eyes (mean age 62.7 ± 12.3 years), RS inter-agreement was moderate for diagnosis (overall κ = 0.59) and referral (κ = 0.47). NRS showed substantial (NRS1 κ = 0.78) to moderate (NRS2 κ = 0.64; NRS3 κ = 0.54) diagnostic agreement and high specificity (> 90% for all). Sensitivity varied across raters and diagnoses (0.57-0.89), with comparatively lower sensitivity observed for certain vision-threatening conditions such as age-related macular degeneration and macular hole. Referral agreement varied (κ = 0.68, 0.24, 0.34) amongst NRS. Visual acuity was the only significant predictor of referral discordance (OR 0.66, 95% CI 0.55-0.80, p < 0.001). Median NRS with PathFinder assistance processing time was < 20s. The most significant false-positive diagnoses made by PathFinder-assisted NRS in eyes deemed normal by RS was ERM (40.0%), followed by PED (20.0%) and AMD (13.3%), observed across a small number of eyes and does not affect the results.

PathFinder is a valuable real-time decision-support tool in resource-limited settings; however, disease-specific refinements and clinical oversight remain important, particularly for vision-threatening conditions.