Importance of model governance in clinical AI models: case study on the relevance of data drift detection

To evaluate the use of data drift detection into an artificial intelligence (AI) governance framework to ensure the safe clinical deployment of an AI model that predicts safe patient discharge after gastrointestinal and oncological surgery. Despite the potential of AI in healthcare, clinical implementation remains limited. Mature AI governance is critical for safe and effective deployment, particularly in dynamic healthcare settings, where patient populations and treatment protocols evolve over time and data drift can occur. This case study illustrates the value of proactive model and data monitoring.

A retrospective evaluation of an AI model that predicts safe discharge after gastrointestinal or oncological surgery was performed using data collected from two centres between June 2017 and October 2022, encompassing 6822 admissions. Data from June 2017 to January 2020 were used for model development, while data from January 2020 to October 2022 were used for temporal validation. Three candidate classification models (random forest, logistic regression and extreme gradient boosting (XGBoost)) were compared based on area under the receiver operating characteristic curve and Brier score. Data drift was monitored using univariate methods (Jensen-Shannon distance and Kolmogorov-Smirnov tests) and a multivariate approach (principal component analysis (PCA) reconstruction error).

Following cross-validation, XGBoost was selected for temporal validation, achieving stable performance over the validation period, with an area under the curve of 0.82 and a Brier score of 0.158. Univariate data drift monitoring detected a significant shift in respiratory rate starting in January 2022, primarily attributed to changes at one hospital. Multivariate analysis using PCA reconstruction error flagged potential data drift in three non-sequential months, with the highest reconstruction error observed in March 2021. Two anomalies were attributed to data entry errors in saturation and heart rate values, while another alert was linked to an unusually long patient length of stay.

By tracking both model performance and data drift, this study identified shifts in data distributions and data quality issues in a clinical AI model. These findings underscore the need for robust model and data governance measures to ensure responsible AI deployment.