Related Experiment Video
Updated: Jan 2, 2026
In Vivo Confocal Microscopy in the Diagnosis and Management of Dry Eye: A Focus on Imaging Protocols and Interpretation
Published on: November 11, 2025
Crowdsourcing Morphology Assessments in Oculoplastic Surgery: Reliability and Validity of Lay People Relative to
Daniel B Rootman1,2, Christine L Bokman1, Blake Katsev1
1Division of Orbital and Ophthalmic Plastic Surgery, Stein Eye Institute, University of California, Los Angeles.
Purpose:
To determine if crowdsourced ratings of oculoplastic surgical outcomes provide reliable information compared to professional graders and oculoplastic experts.
Methods:
In this prospective psychometric evaluation, a scale for the rating of postoperative eyelid swelling was constructed using randomly selected images and topic experts. This scale was presented adjacent to 205 test images, including 10% duplicates. Graders were instructed to match the test image to the reference image it most closely resembles. Three sets of graders were solicited: crowdsourced lay people from Amazon Mechanical Turk marketplace, professional graders from the Doheny Image Reading Center (DIRC), and American Society of Ophthalmic Plastic and Reconstructive Surgery surgeons. Performance was assessed by classical correlational analysis and generalizability theory.
Results:
The correlation between scores on the first rating and the second rating for the 19 repeated occurrences was 0.60 for lay observers, 0.80 for DIRC graders and 0.84 for oculoplastic experts. In terms of inter-group rating reliability for all photos, the scores provided by lay observers were correlated with DIRC graders at a level of r = 0.88 and to experts at r = 0.79. The pictures themselves accounted for the greatest amount of variation among all groups. The amount of variation in the scores due to the rater was highest in the lay group at 25%, and was 20% and 21% for DIRC graders and experts, respectively.
Conclusions:
Crowdsourced observers are insufficiently precise to replicate the results of experts in grading postoperative eyelid swelling. DIRC graders performed similarly to experts and present a less resource-intensive option.
