E pluribus unum: prospective acceptability benchmarking from the Contouring Collaborative for Consensus in Radiation...

Contouring Collaborative for Consensus in Radiation Oncology (C3RO) is a crowdsourced challenge engaging radiation oncologists across various expertise levels in segmentation. An obstacle to artificial intelligence (AI) development is the paucity of multiexpert datasets; consequently, we sought to characterize whether aggregate segmentations generated from multiple nonexperts could meet or exceed recognized expert agreement.

Participants who contoured $$ region of interest (ROI) for the breast, sarcoma, head and neck (H&N), gynecologic (GYN), or gastrointestinal (GI) cases were identified as a nonexpert or recognized expert. Cohort-specific ROIs were combined into single simultaneous truth and performance level estimation (STAPLE) consensus segmentations. $$ ROIs were evaluated against $$ contours using Dice similarity coefficient (DSC). The expert interobserver DSC ( $$ ) was calculated as an acceptability threshold between $$ and $$ . To determine the number of nonexperts required to match the $$ for each ROI, a single consensus contour was generated using variable numbers of nonexperts and then compared to the $$ .

For all cases, the DSC values for $$ versus $$ were higher than comparator expert $$ for most ROIs. The minimum number of nonexpert segmentations needed for a consensus ROI to achieve $$ acceptability criteria ranged between 2 and 4 for breast, 3 and 5 for sarcoma, 3 and 5 for H&N, 3 and 5 for GYN, and 3 for GI.

Multiple nonexpert-generated consensus ROIs met or exceeded expert-derived acceptability thresholds. Five nonexperts could potentially generate consensus segmentations for most ROIs with performance approximating experts, suggesting nonexpert segmentations as feasible cost-effective AI inputs.