Establishing prospective performance monitoring for real-world implementation of deep learning-based...

Deep-learning auto-segmentation (DLAS) performance in radiotherapy may change over time due to data shift/drift or practice changes, yet guidance for quality assurance is lacking. This study developed a practical framework for prospective performance monitoring using retrospective data.

A total of 464 prostate cases over 20 months were retrospectively collected. Two commercial DLAS models were clinically used: model A (2D U-Net, January 2022-January 2023) and model B (3D U-Net, February-August 2023). The agreement between DLAS and clinical contours was assessed using Dice Similarity Coefficient (DSC), 95th percentile Hausdorff Distance (HD95), and Surface DSC with a 2 mm tolerance (SDSC). Statistical process control charts were created to monitor performance drift and model switching. The first 150 cases were used to define organ-specific control limits with two and three standard deviations of monthly mean values, ${\sigma }_{\overline{x}}$ .

2 ${\sigma }_{\overline{x}}$ and 3 ${\sigma }_{\overline{x}}$ -based control limits were established for the monthly average charts, ranging from DSC 0.82-0.97, HD95 1.4-10.5 mm, and SDSC 0.45-0.91 across organs. Model A showed stable performance, with 9-13 months per organ remaining within the 3 ${\sigma }_{\overline{x}}$ thresholds. In contrast, model B demonstrated a marked performance shift (p < 0.001), with all five organs exceeding both thresholds across all 7 months. The 2 ${\sigma }_{\overline{x}}$ thresholds were more sensitive in detecting mild deviations for model A, while both limits effectively identified the substantial drift of model B.

The monitoring system effectively detected out-of-distribution outliers and clinical practice changes, providing a reliable framework for early detection of monthly performance degradation.