A Virtual Reading Center Model Using Crowdsourcing to Grade Photographs for Trachoma: Validation Study

Area of Science:

• Ophthalmology
• Public Health
• Digital Health

Background:

• Trachoma elimination efforts are challenged by declining grader expertise in identifying active disease (trachomatous inflammation-follicular [TF]).
• Accurate TF grading is crucial for public health decisions regarding continued or reinstated treatment strategies.
• Telemedicine for trachoma surveillance requires reliable image interpretation, which is hindered by poor connectivity in endemic regions.

Purpose of the Study:

• To develop and validate a cloud-based virtual reading center (VRC) model utilizing crowdsourcing for interpreting trachoma images.
• To assess the accuracy and efficiency of this VRC model in identifying TF.

Main Methods:

• The Amazon Mechanical Turk (AMT) platform recruited lay graders to interpret 2299 images from a smartphone-based camera system.
• Crowdsourcing scores were aggregated, and an optimal cutoff was determined to maximize kappa agreement and TF prevalence estimation.
• A tiered approach with skilled overread of positive cases was implemented to enhance accuracy.

Main Results:

• Over 16,000 grades were obtained in about an hour for $1098.
• The VRC achieved 95% sensitivity and 87% specificity for TF in the training set (kappa=0.797).
• With skilled overreads, specificity improved to 99%, reducing the grader burden by over 80% and achieving a kappa of 0.685.

Conclusions:

• A VRC model employing crowdsourcing followed by skilled grading of positive images can rapidly and accurately detect TF in low-prevalence settings.
• This model supports further validation for trachoma surveillance and prevalence estimation using field-acquired images.
• Prospective field testing is necessary to confirm the diagnostic acceptability of this VRC model in real-world, low-prevalence surveys.