Community medical centers struggle to produce well-calibrated clinical prediction models: Data augmentation can help

Machine learning models (ML) often require localization to perform optimally in local populations. We hypothesize that smaller community healthcare centers may not have the necessary patient volume to facilitate localization based on statistical guidelines. This work investigates the ability for community medical centers to localize ML and performs a simulation study to evaluate synthetic data generation (SDG) to augment local data for recalibration.

We conducted an experiment using data from a real network of hospitals (two rural, one urban academic medical center) to predict 30-day unplanned hospital readmission and using data from a multi-site ICU dataset to simulate using synthetic data generation (SDG) in a network of hospitals of various sizes. We also performed a simulation study using data from a multi-site ICU dataset to evaluate the utility of SDG to augment local data volumes.

In the real-world evaluation, the urban medical center met the guidelines for the number of samples for recalibration (Required: 14,224, Available: 42,303) and had the best calibrated model using local data (α=0.1,β=1.05; best: α=0,β=1). For the smaller sites, neither site had the samples required for recalibration (Site 1: Required: 16461, Available: 3187; Site 2: Required: 15299, Available: 905). In the simulation study, deep learning-based SDG was most effective at improving calibration performance.

Connections to large medical centers are not enough to promote accurate ML at all sites within a healthcare system. Data augmentation and SDG may provide the necessary data volumes to enable local recalibration at smaller facilities.