Detecting Performance Drift in AI Models for Medical Image Analysis Using CUSUM Chart

The use of artificial intelligence (AI) models to assist clinical decisions for diagnosis and prediction has been shown to improve patient outcomes and clinical decision-making. However, the performance of an AI model deployed in a clinical setting can vary over time or between initial evaluations and clinical use because of data drift. Monitoring the performance of a deployed AI model may help to (i) ensure the AI model performs as expected in the clinical environment and (ii) detect performance deviations and alert stakeholders to these deviations. In this study, we investigate how a change in the performance of an AI model caused by an abrupt data drift can be detected using a cumulative sum (CUSUM) control chart. We demonstrate the use of CUSUM for computer-aided breast cancer detection using data from the publicly available Emory Breast Imaging Dataset (EMBED). Our results indicate that, when the magnitude of the drift is 1.5 times the standard deviation of the performance metric, CUSUM is able to detect changes in the test negativity rate of an AI model within an average of 5 days following the onset of performance drift, with a long duration ( $\sim$ 293 days) between false alarms. Our results also indicate that when the average number of samples is 40/day, leading to a reasonable standard deviation for the performance metric, almost no false alarms are expected in a period of 60 days, with the choice of CUSUM parameters that lead to a small true alarm detection delay. We also demonstrate that CUSUM can be applied for monitoring the performance of AI models even when ground-truth labels are not available. We evaluate the robustness of the CUSUM chart to non-normal distributions and show that when tuned to detect relatively small parameter shifts, the CUSUM chart can be robust to non-normal distributions. We demonstrate that the sensitivity of CUSUM can be controlled to balance the mean time between false alarms and the delay in detecting a true change in performance. We conclude that CUSUM, which is a well-studied statistical process control method, can easily be adapted for monitoring the performance of AI models targeted for assisting in medical diagnosis.