Statistical process control for performance monitoring and continuous quality assurance of deep learning...

After clinical implementation of deep learning segmentation (DLS) models it is highly recommended to perform routine quality assurance (QA). Through forthcoming regulations of the EU AI Act state that the output of any DLS model needs to be logged, enabling continuous QA (CQA) to monitor DLS performance and introduce alarms. Therefore, the goal was to implement a CQA framework for DLS in radiotherapy.

The direct output of the DLS models and the clinically approved delineations (CS) were automatically exported, after which geometric metrics were calculated. For each combination between a region of interest (ROI) and metric, a target, lower and upper control limit were determined, based on statistical process control (SPC). Adapted versions of the first three Nelson rules were used for outlier identification and detection of trend shifts and drifts.

In the first six months, 545 DLS and corresponding CS RT structure files were logged containing 3093 ROIs. From these ROIs, 3.0 % was automatically reported as outlier. Data from 5 patients, which anatomy was deemed interesting, was saved for potential model re-training. Four trend shifts identified a performance drop for the humerus due to unexpected changes. Twelve other trend shifts and one trend drift were detected, causing only temporary deviations.

A CQA framework for DLS was successfully implemented using SPC and adapted Nelson rules to automatically report outliers and trend shifts, which led to automatic detection of such deviations.