Clinical evaluation and regression test of a commercial deep-learning auto-segmentation model

Commercial deep-learning segmentation (DLS) tools are increasingly used in clinical practice. Software updates may alter segmentation performance, highlighting the need for systematic clinical evaluation before implementation.

This study presents our experience in clinically evaluating and regression-testing a RayStation DLS model with the aim of guiding commissioning and routine quality assurance of commercial DLS tools in clinical environments.

U-Net DLS model for normal-tissue structures, originally commissioned in RayStation version 11B, was regression-tested after upgrading to 2024A. Previously commissioned CT datasets for head and neck (n = 18), thorax (n = 18), male pelvis (n = 18), and breast (n = 21) were re-evaluated in RayStation 2024A for regression testing following the software upgrade, while 20 new abdominal cases were included for initial commissioning. Clinical segmentations served as the reference standard. Geometric performance was evaluated using Dice Similarity Coefficient (DSC), 95% Hausdorff Distance (HD95), and Mean Surface Distance (MSD). Dosimetric evaluation used the original clinical treatment plans to compare Dmean and Dmax between DLS and clinical contours, normalized to prescription dose; breast was excluded from dosimetric analysis because its DLS model was not commissioned for clinical use.

Regression testing showed that, in comparison with the previously commissioned data, mean [± standard deviation (SD)] DSC values were 0.81 ± 0.09 for the head and neck, 0.84 ± 0.13 for the thorax, and 0.89 ± 0.06 for male pelvis. Corresponding MSD values were 0.83 ± 0.34 mm, 2.47 ± 1.81 mm, and 1.76 ± 0.68 mm; and HD95 values were 4.75 ± 2.38 mm, 8.45 ± 3.38 mm, and 13.21 ± 7.05 mm. Significant paired differences between versions were limited to selected structures, including the mandible, cochlea, and eye in head and neck; lungs and esophagus in thorax; and prostate in the male pelvis (p < 0.05). Breast segmentation results showed low performance (DSC 0.66 ± 0.15, MSD 4.8 ± 1.93 mm, and HD95 22.2 ± 9.6 mm) compared to clinical reference, so the breast structure models are not recommended for clinical use. Abdominal structures (RayStation 2024A vs clinical) had DSC 0.95 ± 0.014; MSD 0.82 ± 0.32 mm; and HD95 5.4 ± 2.9 mm. The mean dose differences between DLS and clinical segmentations were 0.03 Gy for the head and neck, -0.92 Gy for the thorax, 0.27 Gy for the male pelvis, and 0.02 Gy for the abdominal sites, and the corresponding maximum dose differences were 10.96 Gy, 13.23 Gy, 3.39 Gy, and 6.31 Gy.

We described a procedure for practical quality assurance of auto-segmentation tools for clinical use after periodic software upgrades. Regression testing of head and neck, thorax, and male pelvis sites showed consistent performance after the software upgrade and the DLS models for abdominal site were commissioned in this study for clinical use; while the DLS models for breast site were not recommended.