Predicting radiology no-shows with machine learning: Methodological pitfalls and practical considerations

Predictive machine learning (ML) models may help reduce radiology appointment no-shows and late cancellations, which disrupt care, reduce operational efficiency, and cause financial losses. However, few studies address the practical challenges of developing robust no-show models across large health systems. This study outlines methodological hurdles we faced in building outpatient radiology no-show prediction models at a large academic medical center.

The dataset includes 334,002 outpatient imaging appointments across multiple imaging sites. EHR data was integrated with external datasets for weather, socioeconomic indicators, and geographic data. Preprocessing included cohort selection, feature engineering, and curation strategies to address class imbalance and leakage. Boosted tree classifiers and other standard algorithms were trained on 2022-2023 data and tested on 2024 data.

Our best models achieved AUC = 0.71, consistent with the low end of prior results. Challenges included ambiguous data definitions, distinguishing cancellations from reschedules, and variable data quality across features and sites. Feature engineering was constrained by predictor granularity, and temporal inconsistencies highlighted the risk of data leakage. Key predictors were phone reminder status, appointment confirmation, and modality, while crime, income, and weather showed little utility.

Building no-show models requires extensive data cleaning, feature engineering, and temporal validation. Real-world data constraints increase risks of flawed models from leakage, inconsistencies, and confounding, particularly for prospective predictions.

Successful deployment of no-show models requires more than algorithmic sophistication. Transparent documentation, rigorous temporal validation, and critical data quality assessments are essential to ensure models are reliable, fair, and deployable.