Narrowing the gap: expected versus deployment performance.

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Predictive Modeling

Background:

• Accurate prediction of future performance is crucial for successful clinical model development.
• Overly optimistic performance estimates can lead to the nonuse of predictive models in real-world settings.
• Recurrent Neural Network (RNN) models are increasingly used for clinical predictions.

Purpose of the Study:

• To evaluate how different data partitioning methods affect internal test performance estimates of RNN models.
• To quantify the optimism (overestimation of performance) when internal test performance is compared to real-world deployment.
• To assess the impact of including older data in training sets on model performance.

Main Methods:

• Utilized data from a Pediatric Intensive Care Unit (2010-2020) for two prediction tasks: ICU mortality and Bi-Level Positive Airway Pressure failure.
• Employed various data partitioning strategies, including longitudinal partitioning (testing on newer data), to create development and test sets.
• Trained deployable RNN models on historical data (2010-2018) and evaluated them on subsequent data (2019-2020) to simulate real-world deployment.

Main Results:

• Longitudinal partitioning methods demonstrated the least optimism, providing more accurate estimates of future performance.
• Including older data in the training dataset did not negatively impact the performance of deployable models.
• Utilizing all available data for model development maximized the benefits of longitudinal partitioning for year-to-year performance assessment.

Conclusions:

• Longitudinal data partitioning is a reliable method for estimating the real-world performance of clinical predictive models.
• The inclusion of older data in training sets is feasible and does not compromise model performance.
• These findings support the development of robust and trustworthy predictive models for clinical applications.