Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example.

Area of Science:

• Machine Learning in Radiology
• Medical Informatics
• Data Science in Healthcare

Background:

• Machine learning (ML) models for predicting diagnostic imaging follow-up are susceptible to model drift over time.
• This drift can reduce the accuracy and reliability of predictions based on outdated data.
• Regularly updating these models is crucial to maintain their effectiveness.

Purpose of the Study:

• To evaluate strategies for addressing model drift in ML models used for predicting diagnostic imaging follow-up.
• To compare data augmentation with more recent data versus retraining new predictive models.
• To assess the performance of updated ML models against a baseline.

Main Methods:

• A retrospective study utilized radiology reports from 2016 (old data) and 2019-2020 (new data).
• Support vector machine and random forest (RF) algorithms were trained using augmented data (new + old) and new data only.
• Model performance was evaluated by comparing recall and precision against a baseline model using McNemar's test.

Main Results:

• A new RF model trained with augmented data demonstrated significantly improved recall (0.80) compared to the baseline model (0.66, P = .04).
• The augmented RF model also achieved comparable precision (0.90) to the baseline (0.86).
• Retraining the baseline model with new data did not yield significant improvements in performance.

Conclusions:

• A newly developed RF model, updated with recent data through augmentation, outperformed a simply retrained baseline model in recall.
• These findings highlight the necessity of regularly assessing and updating ML models with current data to mitigate model drift.
• Data augmentation with recent data is an effective strategy for improving ML model performance in radiology.