Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP

Area of Science:

• Ophthalmology
• Medical Imaging
• Artificial Intelligence

Background:

• Retinopathy of prematurity (ROP) is a significant cause of vision loss in premature infants.
• Detecting plus disease, a severe ROP indicator, is challenging due to subjective and time-intensive retinal examinations.
• Existing artificial intelligence (AI) algorithms for ROP detection need validation in diverse populations.

Purpose of the Study:

• To validate ROP.AI, an AI algorithm trained on a single cohort, for detecting plus disease in a multicenter Australian cohort.
• To assess the performance of ROP.AI in a real-world clinical setting across multiple tertiary centers.
• To determine the generalizability of ROP.AI for identifying severe retinopathy of prematurity.

Main Methods:

• Retinal images from routine ROP screening (May 2021-February 2022) across five Australian centers were collected.
• Images were analyzed using ROP.AI, and its diagnostic output was compared against expert ophthalmologist diagnoses.
• Performance metrics including sensitivity, specificity, and area under the receiver operator curve (AUC) were calculated.

Main Results:

• A total of 8052 images were analyzed.
• ROP.AI achieved an AUC of 0.75 for diagnosing plus disease.
• Optimized operating point yielded 84% sensitivity, 43% specificity, and 96% negative predictive value for plus disease detection.

Conclusions:

• ROP.AI successfully detected plus disease in an external, multicenter cohort, validating its performance beyond its initial training data.
• The algorithm demonstrated applicability in real-world conditions without image preprocessing or augmentation.
• Further training could enhance ROP.AI's generalizability for widespread clinical implementation in ROP screening.