Implications of non-stationarity on predictive modeling using EHRs

Area of Science:

• Health Informatics
• Machine Learning in Healthcare
• Clinical Data Science

Background:

• Electronic Health Records (EHRs) generate vast clinical data.
• EHR data is subject to non-stationarity due to evolving medical practices and technology.
• This evolving data landscape challenges the reliability of predictive models.

Purpose of the Study:

• To investigate the impact of non-stationarity on predictive modeling using EHR data.
• To assess how data splitting strategies affect model evaluation under non-stationarity.
• To evaluate model performance for predicting delayed wound healing in outpatient settings.

Main Methods:

• Utilized a large EHR dataset of over 150,000 wounds from 59,958 patients.
• Manipulated the degree of non-stationarity by altering training and testing data splits.
• Compared the performance of various predictive models, including complex and simple classifiers.

Main Results:

• Non-stationarity led to divergent conclusions about model predictive power and calibration.
• The performance advantage of complex models like stacking diminished under non-stationarity.
• Simple classifiers performed comparably to complex methods in the presence of significant data shifts.

Conclusions:

• Failure to account for non-stationarity in EHR data can result in incorrect model selection.
• Model evaluation strategies must consider temporal data dynamics for accurate assessment.
• Careful consideration of non-stationarity is crucial for reliable clinical predictive modeling.