Improving Consensus Scoring of Crowdsourced Data Using the Rasch Model: Development and Refinement of a Diagnostic...

Area of Science:

• Ophthalmology
• Medical Imaging
• Computational Biology

Background:

• Diabetic retinopathy (DR) is a primary cause of vision loss in working-age adults globally.
• Current DR screening methods are effective but underutilized, necessitating innovative approaches for detection.
• Crowdsourcing offers a potential avenue for increasing the accessibility and efficiency of DR screening.

Purpose of the Study:

• To evaluate the diagnostic accuracy of crowdsourced gradings of retinal fundus photographs compared to expert grading.
• To determine if regression methods can enhance the consensus accuracy of crowdsourced classifications beyond simple majority vote.

Main Methods:

• 1200 retinal images from the Messidor dataset were classified by 10 trained Amazon Mechanical Turk (AMT) workers each.
• Majority Vote (MV) consensus was established if half or more workers deemed an image abnormal.
• Rasch analysis was employed to calculate worker ability scores, which were then used as weights in a logistic regression model to improve consensus grading accuracy.

Main Results:

• Majority Vote grading achieved 75.5% accuracy, 75.5% sensitivity, 75.5% specificity, and an AUROC of 0.75.
• A logistic regression model incorporating Rasch-weighted worker scores yielded a significantly higher AUROC of 0.91 (95% CI 0.88-0.93).
• Optimizing for 90% sensitivity resulted in 77.5% overall accuracy with 90.3% sensitivity and 68.5% specificity.

Conclusions:

• Crowdsourced interpretation of retinal images provides a rapid and accurate method for DR screening when compared to gold-standard grading.
• Employing logistic regression with Rasch analysis to weight classifications by worker ability substantially improves the accuracy of aggregated diagnostic grades over simple majority vote.