Leveraging Clinical Time-Series Data for Prediction: A Cautionary Tale

Area of Science:

• Healthcare data science
• Clinical informatics
• Predictive modeling

Background:

• Patient risk stratification models are vital in healthcare, often utilizing time-series data from electronic health records.
• Data extraction for clinical prediction tasks involves various formulations based on prediction time and horizon.
• The chosen formulation can significantly influence model performance and clinical applicability.

Purpose of the Study:

• To investigate the impact of different data extraction formulations on patient risk stratification model performance and clinical utility.
• To demonstrate the necessity of using an outcome-independent reference point for model evaluation.
• To compare the performance of outcome-independent versus outcome-dependent schemes in clinical prediction tasks.

Main Methods:

• Utilized a publicly available Intensive Care Unit (ICU) dataset.
• Focused on two clinical prediction tasks: in-hospital mortality and hypokalemia.
• Employed an outcome-independent evaluation scheme and compared it with an outcome-dependent scheme.

Main Results:

• The formulation of data extraction significantly impacts model performance and clinical utility.
• An outcome-independent evaluation scheme is necessary to avoid unrealistic performance metrics.
• The outcome-independent scheme outperformed the outcome-dependent scheme for both in-hospital mortality (AUROC .882 vs. .831) and serum potassium prediction (AUROC .829 vs. .740).

Conclusions:

• The choice of prediction time and horizon in data extraction is critical for developing effective patient risk stratification models.
• Evaluating models using an outcome-independent reference point is essential for realistic performance assessment.
• Outcome-independent data extraction schemes offer superior performance and clinical utility compared to outcome-dependent schemes in real-world healthcare applications.