The Use of Crowdsourcing Technology to Evaluate Preoperative Severity in Patients With Unilateral Cleft Lip in a...

Crowd sourcing has been used in multiple disciplines to quickly generate large amounts of diverse data. The objective of this study was to use crowdsourcing to grade preoperative severity of unilateral cleft lip phenotype in a multiethnic cohort with the hypothesis that crowdsourcing could efficiently achieve similar rankings compared to expert surgeons. Deidentified preoperative photos were collected for patients with primary, unilateral cleft lip with or without cleft palate (CL ± P). A platform was developed with C-SATS for pairwise comparisons utilizing Elo rankings by crowdsource workers through Amazon Mechanical Turk. Images were independently ranked by 2 senior surgeons for comparison. Seventy-six patients with varying severity of unilateral (CL ± P) phenotype were chosen from Operation Smile missions in Bolivia, Madagascar, Vietnam, and Morocco. Patients were an average of 1.2 years' old, ranging from 3 months to 3.3 years. Each image was compared with 10 others, for a total of 380 unique pairwise comparisons. A total of 4627 total raters participated with a median of 12 raters per pair. Data collection was completed in <20 hours. The crowdsourcing ranking and expert surgeon rankings were highly correlated with Pearson correlation coefficient of R = 0.77 (P = 0.0001). Crowdsourcing provides a rapid and convenient method of obtaining preoperative severity ratings, comparable to expert surgeon assessment, across multiple ethnicities. The method serves as a potential solution to the current lack of rating systems for preoperative severity and overcomes the difficulty of acquiring large-scale assessment from expert surgeons.