Diagnostic framework to validate clinical machine learning models locally on temporally stamped data

Area of Science:

• Clinical Informatics
• Machine Learning in Healthcare
• Oncology Data Science

Background:

• Real-world oncology environments exhibit high dynamism due to evolving practices, technologies, and patient demographics.
• This variability can lead to data shifts, negatively impacting machine learning model performance.
• Existing diagnostic frameworks for vetting clinical machine learning models for temporal consistency are limited.

Purpose of the Study:

• To introduce a model-agnostic diagnostic framework for validating clinical machine learning models using time-stamped data.
• To assess the temporal consistency and future applicability of machine learning models in dynamic medical settings.

Main Methods:

• A cohort of over 24,000 cancer patients receiving antineoplastic therapy was analyzed using EHR data from 2010-2022.
• Three models (LASSO, RF, XGBoost) were implemented within a four-stage validation framework.
• The framework included data partitioning, temporal evolution characterization, model longevity analysis, and feature importance assessment.

Main Results:

• The framework identified fluctuations in features, labels, and data values over time when predicting acute care utilization (ACU) in cancer patients.
• Performance was evaluated by partitioning multi-year data into training and validation sets.
• Feature importance and data valuation algorithms were used for data quality assessment and feature reduction.

Conclusions:

• Data timeliness and relevance are critical for the effective deployment of machine learning models in clinical practice.
• The study demonstrated moderate signs of data drift in predicting ACU, underscoring the importance of temporal considerations.
• The developed framework is crucial for validating machine learning models at the point of care.