A Reproducible COCO-Polygon Quality-Control Pipeline Improves Segmentation Stability in Endoscopic Airway Imaging

Background/Objectives: Endoscopic airway imaging, used in endoscopy-guided airway management, enables geometry-based assessment of the airways. However, COCO-format datasets often include fragmented regions and geometrically inconsistent polygon annotations. Such inconsistencies may reduce reproducibility and spatial stability in deep learning-based image segmentation. This study proposes a systematic annotation quality-control (QC) workflow to improve dataset integrity before model training. Methods: The phantom subset of the Upper Airway Anatomical Landmark (UAAL) dataset containing 4526 polygon instances across 2746 frames (2267 training; 479 validation) was analyzed. The QC pipeline validated polygon structure, generated masks at native image resolution, and removed small noise-like instances using an area threshold. A YOLOv8-seg model was trained using (i) original annotations and (ii) QC-refined annotations. Performance was evaluated using precision, recall, mAP@0.5, mAP@0.5:0.95, and Dice similarity coefficient (DSC). Frame-level DSC values were compared using the Wilcoxon signed-rank test. Results: Annotation QC improved boundary consistency and reduced mask fragmentation. Training with QC-refined annotations increased box mAP@0.5:0.95 from 0.602 to 0.628 and mean DSC from 0.823 to 0.830 (p < 0.05). A pilot evaluation on the UAAL clinical subset also showed improved performance, with Box mAP@0.5 increasing from 0.635 to 0.706 and Mask mAP@0.5 from 0.631 to 0.704. Conclusions: Annotation-level QC enhances segmentation robustness without modifying the network architecture. The proposed workflow improves the reproducibility of results in endoscopic image segmentation and may support more stable geometry-based airway analysis in deep learning applications.