From Subjectivity to Reliability: A Multi-Rater Framework for Ground-Truth Generation in Laryngeal Spatial Segmentation

Spatial segmentation is a critical component of objective analysis across a wide range of laryngeal imaging modalities. Manual segmentation is the current gold standard for evaluating and training automated methods, which is inherently subjective and susceptible to variability. However, systematic evaluations of this variability and rigorous methods for generating reliable ground truth are limited, as most prior work has relied on a single rater. This study used spatial segmentation of high-speed videoendoscopy (HSV) recordings as a test case to (1) quantify differences among expert segmentations of the vocal fold edges and assess their reliability, highlighting the need to incorporate multiple raters and report inter- and intra-rater reliability scores, and (2) introduce and evaluate a multi-stage framework for generating highly reliable ground truth with sub-pixel resolution.

Twelve HSV recordings were segmented by three experts using a proposed iterative, three-stage framework that included initial segmentation, reconciliation of high-discrepancy regions, and consensus review. Reliability was assessed using a novel metric of maximum edge variability, and linear mixed-effects models examined the effects of rater, stage, and number of raters on segmentation reliability and ground-truth uncertainty.

Experts differed substantially in initial segmentations with a large effect size, confirming the need for multiple raters. The experiments demonstrated that increasing the number of raters significantly improved ground-truth reliability and reduced uncertainty, with the largest gain from one to two raters. The proposed framework markedly enhanced both intra- and inter-rater reliability, with the most significant improvement occurring during the consensus stage. The proposed maximum edge variability provided a robust alternative to Intersection over Union (IOU), addressing IOU's sensitivity to glottal area size.

Manual segmentation demonstrated considerable variability, reinforcing the need for multi-rater protocols and explicit reporting of reliability evaluation. The proposed framework offers a systematic, unbiased approach for generating reliable ground truth with sub-pixel resolution that is readily applicable to other laryngeal imaging modalities (eg, videostroboscopy, Narrow Band Imaging) where consistent boundary delineation and explicit uncertainty quantification are essential for valid measurement and assessment.