Distribution Shift Analysis in Generalizable Modelling: Intensive Care Time-Series Data.

Area of Science:

• Medical Informatics
• Machine Learning
• Critical Care Medicine

Background:

• Model generalizability is challenged by distribution shifts in medical time-series data.
• Intensive care unit (ICU) data presents unique non-stationary and multivariate characteristics.
• Accurate prediction of critical events like hypotension and hypoxemia is vital.

Purpose of the Study:

• To investigate the impact of distribution shift on model generalizability in ICU time-series data.
• To define and quantify clinically relevant domain adaptation scenarios.
• To evaluate methods for measuring different types of distribution shifts.

Main Methods:

• Utilized the MIMIC-III Matched Waveform Database v1.0 for hypotension and hypoxemia prediction.
• Employed the Kolmogorov-Smirnov (KS) test on a lower-dimensional data representation to quantify distribution shift.
• Defined domain adaptation scenarios reflecting real-world clinical challenges.
• Correlated KS test results with Root Mean Square Error (RMSE) differences for performance evaluation.

Main Results:

• Highlighted challenges in measuring covariate, prior probability, and semantic shifts in complex time-series.
• Observed strong linear correlations (0.74 and 0.99) between shift quantification and model performance degradation.
• Demonstrated the relationship between distribution shift metrics and out-of-distribution performance.

Conclusions:

• Distribution shift is a critical factor affecting the generalizability of predictive models in intensive care.
• Accurate quantification of modality-specific data shifts is essential for robust model development and benchmarking.
• Future research should focus on methods that explicitly address these data shift characteristics.