Beyond the Model: Evaluating AI Decision Strategies for Diabetic Retinopathy Screening Under Clinical Constraints

Background Diabetic retinopathy is one of the leading causes of preventable blindness worldwide, yet it can be stopped through early detection. AI models are increasingly being used as a key enabler to automate this screening, and the results in research settings look very promising. The real challenge is not building more sophisticated models but deploying one that works safely in real clinics. Clinical safety standards require the system to catch nearly every true case of disease, even if that means sending many healthy patients for unnecessary follow-up. This trade-off between keeping patients safe and avoiding a flood of false alarms is the core problem this paper addresses. Given that clinics typically fix a minimum sensitivity target in advance, this study compares decision-making strategies held to the same sensitivity requirement to determine which produces the fewest unnecessary referrals.  Methods We evaluate five decision strategies under identical conditions on the public EyePACS dataset of 5,270 retinal fundus images, with 1,366 labeled as having diabetic retinopathy. The strategies range from a single AI model making every referral decision independently, to ensemble methods that combine the probability scores of multiple models into one unified output, to a two-tiered method in which all images are first screened by a primary model, and a group of secondary models can overturn a referral when their disagreement is high enough. Each strategy is evaluated under two clinically grounded sensitivity targets, a strict 95% requirement and a more moderate 90% requirement, so the results reflect realistic deployment conditions rather than unconstrained optimal performance. Results When the system is required to achieve 95% sensitivity, all strategies produce high false-positive rates, with the best single model reaching only 17.5% specificity. Ensembles offer only marginal improvement at this threshold, while majority voting consistently performs worst across both sensitivity levels. Reducing the sensitivity target from 95% to 90% alone decreases false positives by about 17%. When this lower threshold is combined with an ensemble strategy, unnecessary referrals drop by nearly 25% compared with a single model at 95%. Neither adjustment alone produces this level of improvement; the benefit appears only when both are applied together. Conclusions This study shows that two decisions matter most in AI-based diabetic retinopathy screening: the sensitivity target a clinic sets and the decision strategy it pairs with that target. Although the sensitivity target had a greater influence on referral burden, the best outcomes occurred only when both the target and the strategy were carefully chosen. Pairing a 90% sensitivity target with a weighted ensemble reduced unnecessary referrals by nearly 25% compared with a single model at the stricter 95% target, while majority voting produced the highest false-positive burden at both thresholds. These findings suggest that clinically grounded threshold selection is just as important as the decision strategy itself and that seemingly intuitive approaches such as majority voting may underperform when evaluated under the same safety constraints.