Customizing AI-based screening with real-world data: Practical insights from diabetic retinopathy

Diabetic retinopathy (DR) is a leading cause of vision loss in middle-aged adults globally. Although artificial intelligence (AI)-based screening tools like IDx-DR (classification) and Thirona RetCAD (regression) have shown high sensitivity in controlled settings, real-world screening faces challenges due to missing or low-quality images and inadequate adaptation to local healthcare needs. The objective was to compare the performance of two AI-based DR screening algorithms (IDx-DR and RetCAD) that analyse non-mydriatic images, against ophthalmologists' mydriatic fundoscopy with image analysis and the impact of customized referral threshold modification ('Greifswald modification') on screening outcomes.

This one-centre observational study included 1716 patients with diabetes mellitus (Clinical Trials Register: DRKS00035967). Sensitivity, specificity, the proportion of ungradable images and the reduction in ophthalmologic evaluations were assessed. Customized referral threshold modification was conducted using the Youden Index.

In 98 patients (5.7%), no images could be acquired, and 35 patients (2.1%) had incomplete image sets for IDx-DR. IDx-DR rejected 438 patients (25.5%) due to image quality, while RetCAD flagged 134 eyes from 120 patients (6.9%) but provided output for all. Among analysable images, sensitivities ranged from 70.4% (RetCAD) to 93.6% (RetCAD with Greifswald modification). Including all patients reduced sensitivity from 52.7% (IDx-DR) to 79.9% (RetCAD with Greifswald modification). AI screening reduced ophthalmologic exam needs by 47.5% to 78.5%.

Real-world DR screening performance of AI algorithms, when including non-analysable patients, can be substantially lower than in controlled studies. The use of regression algorithms enabled the customization of referral thresholds, improving screening accuracy and reducing the clinical burden.