Head-to-Head Comparative Evaluation of Four Commercially Available Artificial Intelligence Systems for Detecting...

Comparative evaluations of commercially available artificial intelligence (AI) systems for use in diabetic retinopathy (DR) screening, particularly studies that identify systems by name, are limited, constraining procurement and implementation. This study aimed to identify commercially available AI systems potentially suitable for DR screening in a low-resource Tanzanian setting and compare their accuracy in detecting referable DR.

Through a scoping review and expert consultation, we identified AI systems potentially suitable for implementation. Systems confirmed as suitable, and whose developers agreed to participate, were evaluated. Performance was assessed on a data set of retinal images collected from a Tanzanian DR screening program. The primary outcomes were sensitivity and specificity in detecting referable DR. Additional implementation data, including regulatory approval, referral thresholds, and additional product features, were also collected.

Four commercially available AI systems (Medios AI/Remidio, MONA, Ophtai, and SELENA+) were evaluated. Among 689 people included in the test data set, 379 (55.0%) had referable DR and 93 (13.5%) had proliferative DR. Sensitivity in detecting referable DR ranged from 83.9% to 93.7%, with lower specificity ranging from 70.3% to 79.0%. Sensitivity in detecting proliferative DR exceeded 98% in all four AI systems. All the evaluated AI systems were Conformité Européenne-marked medical devices; one system (Medios AI/Remidio) functions offline as standard.

Several commercially available AI systems demonstrated high sensitivity in detecting referable and proliferative DR, supporting their potential implementation. Consensus on minimum performance thresholds and consideration of implementation factors such as regulatory approval and offline functionality are needed.